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Abstract 


The  estimate  of  software  development  coat  is  a key  piece  of  information 
in  many  software  management  decisions,.  No  technique  exists  which  can 
consistently  produce  the  reliable  and  accurate  cost  estimates  which  managers 
need.  This  thesis  research  effort  explored  the  software  cost  estimating 
process  at  the  Electronic  Systems  Division  (ESD)  of  the  Air  Force  Systems 
Command.  The  purpose  of  the  research  was  to  provide  managers,  researchers, 
and  cost  estimators  with  a better  insight  into  the  cost  estimating  process. 

Data  were  gathered  from  16  major  software  acquisitions  at  ESD  using 
both  a. structured  interview  and  contractor  furnished  Cost  Performance 
Reports.  The  research  findings  identified  some  major  problems  which  are 
currently  inhibiting  the  development  of  accurate  and  reliable  software 
cost  estimates. 

To  reduce  these  problems,  recommendations  are  made  to  adopt  a common 
cost  estimating  technique  and  to  modify  the  use  of  contractor  furnished 
/software  cost  information.  While  the  research  was  limited  to  ESD,  the 
research  findings  and  the  recommendations  may  be  applicable  to  other  DoD 
software  acquisition  agencies. 
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AN  EXPLORATORY  STUDY  OF  SOFTWARE  COST  ESTIMATION  AT 
THE  ELECTRONIC  SYSTEMS  DIVISION 

I*  Introduction 

DoD  program  managers  will  buy  an  estimated  three  billion  clollax-o  worth 
of  software  in  19?6  (Aviation  Week,  1976:4-1)*  Other  managers  throughout 
the  United  States  will  buy  an  additional  fifteen  billion  dollars  worth  of 
software  (Boehm,  1975s 4),  These  managers  face  a common  problem  in  pre- 
dicting or  estimating  the  cost  of  software* 

In  general,  the  accuracy  of  software  cost  estimates  has  been  poor. 
Underestimation  by  a factor  of  2,5  is  common  while  overeatimationa  ars 
unheard  of  (Schwartz,  1975:56),  While  a number  of  effortB  have  been  made 
to  develop  improved  cost  estimation  techniques,  no  generally  accurate  nor 
reliable  method  for  estimating  software  development  costs  has  been  found 
(Morin,  1974). 

Objective 

The  objective  of  this  research  effort  is  to  describe  the  nature  and 
status  of  the  software  cost  estimating  environment. 

Purpose 

The  purpose  of  this  research  is  to  provide  managers,  researchers, 
and  cost  estimators  a hotter  insight  into  the  software  cost  estimating 
problem.  It  is  hoped  that  this  increased  insight  into  the  problem  area 
will  ultimately  result  in  improved  software  cost  estimation. 
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This  research  effort  is  Halted  to  describing  ths  software  cost 
estlaatlng  enrirounsnt  in  the  Electronic  Systems  Division  (BSD)  of  the 
Air  Force  Sysfcm*  Command.  BSD  Is  located  at  Hanscom  AFB  in  Massachusetts 
and  is  responsible  for  a major  portion  of  Air  Force  software  acquisitions. 
While  software  in  also  procured  at  other  Air  Force  locations,  the  ten 
weeks  available  for  this  research  effort  dictated  that  only  th'  ESD 
environment  could  be  explored. 

In  addition , the  research  is  Halted  to  those  major  software  acquisi- 
tions at  BSD  whlch  are  currently  proposed,  are  under  way,  or  are  recently 
completed.  Til*  limitation  was  necessary  due  to  the  general  non-avail- 
ahility  of  data  concerning  past  programs  and  limited  the  data  sources  to 
twenty-one  major  software  acquisitions. 

No  classified  data  was  collected  during  the  research  effort.  Howevor, 
this  had  no  impact  upon  the  research  due  to  the  unclassified  nature  of  th® 
data  which  weu  sought. 

One  last  limitation  on  this  research  effort  is  caused  by  the  method- 
ology which  w«s  selected.  The  data  were  largely  collected  using  a 
structured  interview.  The  limitations  of  this  technique  are  discussed  in 
Chapter  h,  Methodology, 

Applicability  of  Research  Findings 

Since  tit  research  Is  limited  only  tc  software  developments  at  ESD, 
the  applicability  of  the  research  findings  to  other  software  developments 
may  reasonably  be  questioned. 
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In  many  ways,  the  ESD  environment  is  unique*  ESD  receives  systems 
engineering  support  from  the  MITRE  Corporation,  a Federal  Contract  Research 
Center  (FCRC),  The  relationship  between  ESD  program  offices  and  MITRE  is 
somewhat  unique  in  the  DoD*  Also,  the  types  of  systems  procured  by  ESD 
ere  primarily  command,  control,  and  communications  systems*  The  relation- 
ship betwoen  these  types  of  software  developments  with  other  types  of 
software  developments  (e*g*  avionics,  inventory  control,  personnel  systems) 
is  not  well  defined. 

However,  despite  the  somewhat  unique  aspects  of  the  ESD  environment, 
there  are  many  facets  of  the  environment  which  are  quite  similar  to  that 
of  other  DoD  software  acquisition  agencies.  The  types  of  contracts 
issued,  the  regulations  covering  system  acquisition,  and  ths  common 
weapon  system  cycle  tend  to  make  the  software  acquisitions  of  ESD  similar 
to  that  of  other  DoD  agenciec.  Also,  the  software  systems  at  ESD  are 
generally  large  software  systems  in  excess  of  >500,000*  The  very  size 
of  the  software  acquisitions  tends  to  produce  many  common  effects  in 
software  development  regardless  of  the  intended  application  of  the  soft- 
ware system. 

Therefore,  while  the  uniqueness  of  the  ESD  environment  may  tend  to 
limit  the  application  of  the  research  findings,  it  appears  reasonable  to 
assume  that  these  findings  may  apply  in  . general  fashion  to  other  large 
software  developments  made  by  other  DoD  agencies. 

Assumptions 

Since  the  research  is  descriptive  in  nature,  no  explicit  assumptions 
are  made  concerning  the  software  cost  estimating  environment  at  ESD, 
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To  date,  most  of  the  research  in  the  area  of  software  cost  estimation 
has  been  in  the  search  for  causative  relationships.  Researchers  are  trying 
to  determine  the  relationship  between  various  software  parameters  and  cost. 
There  have  beer  no  published  efforts  which  have  described  the  envlronment- 
in  which  coat  estimates  are  made  and  uaed.  The  writer  boll even  that  euch 
a research  effort  has  value  to  three  types  of  people. 

The  Manager . The  manager  of  a software  development  makes  a number  of 
decisions  which  rely  on  a software  cost  estimate.  To  make  these  decisions, 
a manager  must  understand  the  data  with  which  he  is  presented.  This 
research  seeks  to  aid  the  manager* a understanding  of  the  software  cost 
estimating  process  so  that  he  may  better  utilize  the  estimates  presented 
to  him. 

Managers  are  frequently  evaluated  on  how  well  budgeted  cost  agrees 
with  actual  cost.  If  a software  project  experiences  a 200%  overrun,  the 
manager’s  performance  may  look  poor.  However,  with  the  current  state  of 
software  cost  estimating,  it  is  difficult  to  determine  whether  the  200% 
overrun  was  due  to  poor  management  or  poor  estimating.  This  research 
effort  hopes  to  assist  managers  in  understanding  why  software  cost  esti- 
mates are  difficult  to  prepare  and  frequently  erroneous. 

The  Researcher.  Research  continues  in  an  effort  to  establish  valid 
cost  estimating  relationships  for  software.  To  date,  these  efforts  have 
met  with  very  limited  success,  A major  objective  of  this  descriptive 
research  is  to  provide  other  researchers  with  clues  to  cause  and  effect 
relationships  which  may  exist.  This  effort  may  help  to  explain  how  and 
why  certain  cost  estimating  techniques  are  used  or  not  used.  Equipped 
with  a bettor  insight  to  the  nature  of  the  software  cost  estimating 
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environment,  other  researchers  may  be  aasisted  In  their  efforts. 

The  Coat  Estimator.  The  cost  estimator  is  faced  with  the  task  of 
assigning  a cost  to  software,  a very  intangible  object.  Other  cost  esti- 
mators need  only  to  measure  the  weight  and  speed  of  an  aircraft  to  reason- 
ably estimate  aircraft  coBta.  However,  the  software  cost  estimator  does 
not  find  hie  job  easy.  He  is  faced  with  numerous  techniques  which  require 
the  use  of  a large  number  of  vaguely  defined  variables.  He  continues  to 
search  for  a better  method,  but  cannot  amass  enough  experience  on  his  own 
to  determine  which  techniques  are  better  than  others  or  which  techniques 
are  easier  to  apply. 

This  research  effort  does  not  develop  a new  coat  estimating  tech- 
nique. It  does  not  determine  causative  relationships.  Instead,  this 
effort  only  seeks  to  describe  the  environment  surrounding  the  software 
cost  estimation  process.  A basic  objective  is  to  communicate  the  experi- 
ences of  some  software  cost  estimators  to  other  software  cost  estimators. 
The  sharing  of  experience  in  this  area  has  been  limited.  This  study  seeks 
to  enhance  the  sharing  of  information. 


Thesis  Outline 

The  thesis  consists  of  six  chapters.  The  first  chapter  has  discussed 
the  nature  of  the  research  effort. 

Decision  Making  and  Software  Coat  Estimates.  Chapter  2 discusses  the 
decisions  made  during  the  weapon  system  acquisition  process  which  rely 
upon  a software  cost  estimate.  These  decisions  range  from  selecting  a 
weapon  system  to  awarding  software  contracts. 

State  of  the  Art  of  Software  Cost  Estimating,  Once  the  reader  is 
familiar  with  the  decisions  which  utilize  a software  cost  estimate, 
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Chapter  3 examines  the  current  state  of  the  art  of  software  cost  estimat- 
ing. To  understand  the  USD  environment,  one  must  first  understand  tho 
state  of  the  art  of  software  cost  estimating  as  presented  in  the  litera- 
ture. Five  general  categories  of  coat  estimating  techniques  are  reviewed. 
The  problems  associated  with  using  these  techniques  or  with  developing 
new  ones  are  also  discussed. 

Methodology.  After  the  background  provided  by  Chapters  2 and  3* 
Chapter  4 details  the  methodology  used  for  the  reaearch  effort.  The 
hypotheses  which  were  initially  formed  as  well  as  the  interview  developed 
to  test  the  hypotheses  are  discussed  in  general. 

Findings.  Chapter  5 contains  the  research  findings.  Each  of  the 
initial  hypotheses  are  reviewed  along  with  the  data  which  was  collected. 
The  data  are  analyzed  and  a determination  is  made  concerning  the  research 


hypotheses. 
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II,  Declgloi 


Before  looking  at  how  software  coat  estimates  are  made.  It  la  impor- 
tant to  understand  how  the  estimate  is  used  in  making  decisions.  This 
chapter  discusses  the  decisions  made  during  the  weapon  system  acquisition 
process  which  are  influenced  by  the  software  cost  estimate. 


Economic  Evaluation  Of  Weapon  Systems 


The  software  development  cost  estimate  is  timed  in  the  economic 
evaluation  of  weapon  system  alternatives.  For  example,  the  Air  Force 
might  be  faced  with  deciding  whether  to  update  an  aging  air  defense 
system  or  to  develop  a new,  advanced  system.  To  make  this  decision, 
decision  makers  must  understand  the  cost  of  the  hardware  and  software 
associated  with  each  alternative. 

While  softy  Are  cost  may  have  been  minor  in  the  past,  it  is  now 
frequently  a major  component  of  total  system  cost.  For  example,  the 
World  Wide  Military  Command  and  Control  System  (WWMCCS)  will  expend  over 
1722  million  on  software  and  lees  than  #100  million  on  hardware  (Air 
Force  OSR,  1973:28).  During  nine  years  of  development,  the  Army  spent 
an  estimated  #467  million  on  software  development  for  the  Safeguard 
System  (Ashe  et  al,  1975:2-13),  Over  a four  year  period,  the  Minuteman 
program  office  spent  $124  million  on  software  (Ashe  et  al,  1975:2-22). 

In  the  1980*8  software  can  be  expected  to  become  an  ever  increasing  portion 
of  weapon  system  cost.  Therefore,  to  evaluate  alternative  weapon  system*, 
a decision  maker  must  have  a reliable  software  cost  estimating  technique. 
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Design  Tradeoffs 

The  eoftwe.ro  coat  estimate  is  also  used  in  staking  engineering  design 
tradeoffs  within  a weapon  system*  Design  engineers  must  decide  whether  to 
select  an  older  computer  with  a large  set  of  support  software,  or  to 
select  a newer  computer  with  more  computational  speed  but  less  support 
software.  They  must  decide  whether  a particular  function  ia  last  performed 
by  software,  by  hardware,  by  firmware,,  or  even  by  manual  procedures,.  They 
must  also  decide  whether  to  "make  or  buy"  software. 

With  the  advent  of  microprocessing,  "off  the  shelf"  software  packages, 
and  other  developments,  the  design  engineer  is  faced  with  a myriad  of 
design  alternatives.  To  select  the  best  alternative,  the  engineer  must 
be  able  to  reliably  estimate  the  software  development  cost  associated 
with  each  alternative. 

Budgeting 

The  software  cost  estimate  forme  the  basis  for  the  budget  or  financial 
plan  of  the  weapon  system.  Errors  In  the  estimate  cause  errors  in  the 
budget.  If  software  costa  are  seriously  underestimated,  a program  manager 
might  have  to  request  additional  funds  from  DoD  or  Congress,  Additional 
funds  might  not  be  readily  available  causing  the  weapon  system  to  slip 
its  schedule  while  awaiting  additional  funds. 

In  general,  the  request  for  additional  funaa  is  quickly  labeled 
"overrun"  and  not  "inaccurate  estimate".  The  program  manager  finds  that 
his  program  is  subjected  to  increased  surveillance  and  that  his  management 
flexibility  is  reduced  (Large,  1974*10). 

Therefore,  to  avoid  the  problems  associated  with  seeking  additional 
funds  and  to  avoid  the  criticism  associated  with  overruns,  a program 
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manager  needs  a reliable  technique  to  estimate  and  budget  software  develop- 
ment cost. 

Competitive  Contract  Award 

The  software  cost  estimate  is  also  a primary  factor  in  the  award  of 
software  development  contracts.  Proposals  from  competing  contractors  are 
analyzed  and  the  contract  is  usually  awarded  to  the  lowest  bidder.  If 
the  lowest  bidder's  price  is  unreasonably  low,  two  problems  are  created. 
First,  the  contract  award  is  inequitable.  Awarding  to  a bidder  who 
has  seriously  underestimated  the  cost  of  a contract  and  is  eventually 
bailed  out  by  the  government  penalizes  other  bidders  who  more  accurately 
understood  and  estimated  the  contract  requirements, 

A second  problem  develops  from  awarding  a contract  at  an  unreasonably 

* 

low  cost.  In  a study  done  by  RAND  Corporation,  their  findings  indicated 
that  underestimating  a contract  leads  to  cost  growti  An  excessively  low 
bid  forces  a contractor  to  make  unwise  decisions  in  an  attempt  to  stay 
within  costs.  For  example,  the  contractor  might  award  to  a less  qualified 
sub-contractor  or  place  insufficient  emphasis  on  the  production,  operation, 
aad  maintenance  aspects  of  a weapon  system.  When  contract  costs  rise,  the 
government  frequently  balls  out  the  contractor  and  is  faced  with  paying 
not  only  the  true  or  expected  cont  of  the  contract,  but  also  the  additional 
costs  caused  by  the  contractor's  unwise  decisions  (Large,  1974), 

Therefore,  in  order  to  equitably  award  a contract  at  a reasonable 
price,  contractors  and  contracting  officers  need  a reliable  technique  of 
estimating  software  costa. 
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Contract  Schedule  and  Manpower 

The  winning  contractor  uses  his  estimate  of  aoftware  development  coat 
to  determine  how  many  men  to  assign  to  a project  and  how  many  montha  to 
allocate  to  aoftware  development*  If  eoftwafre  coata  are  underestimated, 
either  too  few  programmers  will  be  aaeigned  to  the  project  or  :oo  abort 
a period  will  be  allocated  for  software  development. 

When  the  contractor  becomes  aware  of  the  true  size  and  coat  of  the 
aoftware,  he  must  either  add  programmers  to  the  project  or  extend  the 
project* a schedule,  Either  of  these  actions  can  create  new  problems*  For 
example,  when  new  programmers  are  added  to  a project,  the  original  pro- 
grammers stop  producing  software  while  they  train  the  new  programmers  and 
develop  a new  software  production  plan.  The  cost  of  this  additional 
training  and  planning  effort  increases  software  cost  even  higher*  Main- 
taining tb'.  existing  schedule  by  adding  people  might  appear  simple,  but  in 
many  cases  the  results  are  disastrous  (Brooks,  1975:24)* 

On  the  other  hand,  extending  the  project*s  schedule  also  creates 
problems.  Most  weapon  systems  are  both  hardware  and  software  developments 
with  software  frequently  on  the  critical  path  of  the  schedule*  While 
waiting  for  a small  software  effort  to  be  completed,  an  entire  multi- 
million  dollar  program  might  be  delayed.  The  indirect  cost  of  software 
due  to  this  type  of  delay  could  be  fifty  times  the  additional  software 
cost  (Findley,  1974:15)*  This  large  indirect  cost  is  due  to  the  signifi- 
cant impact  software  can  have  upon  the  schedule  or  performance  of  a 
weapon  system* 

Another  less  obvious  problem  can  result  from  providing  inadequate 
time  or  manpower  to  a software  project*  When  the  software  project  manager 
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find*  himself  vunni ag  short  of  tins  and  manpower,  on*  possible  action  he 
raight  take  is  to  reduce  the  quality  of  the  software  product*  tie  sight 
encourage  inefficient  programing,  less  testing  or  poorer  documentation 
in  an  effort  to  remain  within  his  inadequate  resources* 

Therefore,  to  avoid  the  problems  associated  with  inadequate  schedulos 
and  manpower,  a manager  needs  a reliable  technique  for  estimating  software 
development  resources* 


Contract  Status 

Another  use  of  the  software  cost  estimate  is  in  monitoring  the  status 
of  software  development*  It  is  difficult  to  establish  clearly  defined 
technical  milestones  for  software.  Because  of  this  difficulty,  a common 
method  of  evaluating  the  technical  status  of  software  development  is  to 
compare  the  contractor's  actual  coat  to  the  contractor's  budgeted  coat* 

For  example,  if  a contractor  has  expended  90%  of  his  budgeted  cost  for 
software,  one  might  infer  that  the  software  is  90%  complete*  However, 
such  an  inference  is  only  correct  if  the  budgeted  cost  of  software  is 
accurate*.  If  the  budgeted  or  estimated  cost  of  software  is  wrong,  then 
the  government  la  only  monitoring  financial  expenditures  and  not  technical 
progress* 

If  valid  status  indications  are  to  be  obtained  for  softwax-e  develop- 
ment, an  improved  set  of  technical  milestones  for  software  as  well  as  a 
reliable  software  cost  estimating  technique  are  required. 

Reasonable  Estimates  And  Good  Decisions 

Cost  estimators  cannot  predict  the  future.  No  crystal  balls  are 
involved.  Instead,  a coat  estimator  tries  to  determine  a "reasonable" 
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value  for  the  resource*  needed  to  carry  out  a particular  plan*  Therefore, 
a decision  maker  cannot  expect  highly  accurate  cost  estimate*  to  support 
his  decisions* 


However,  all  of  the  previously  discussed  decisions  do  not  require 
highly  accurate  estimates,  Kout  of  these  decisions  could  probably  be  made 
reasonably  well  with  an  estimate  that  varied  tuu  much  an  plus  or  minus  25%, 
However,  as  we  shall  see  in  later  chapters,  current  techniques  available 
for  estimating  software  cost  frequently  have  errors  in  excess  of  25C %* 

With  errors  of  this  magnitude,  making  good  decisions  concerning  software 
developments  becomes  increasingly  difficult  If  not  Impossible* 

If  we  want  to  make  good  decisions  concerning  software,  we  need  to  be 
able  to  reasonably  estimate  software  costs.  If  we  want  to  make  tradeoffs 
between  systems  and  within  a system,  if  we  want  to  award  contracts 
equitably  and  carry  them  out  successfully,  then  decision  makers  must  have 
a reliable  technique  for  estimating  software  development  cos'-.  With 
these  decisions  in  mind,  we  can  now  look  at  the  state  of  the  art  of  soft- 
ware cost  estimating  and  examine  the  techniques  which  are  currently 
available. 
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III*  State  Of  The  Art  Of  Software  Coat  Estlmat:^ 

Earing  seen  how  managers  and  engineers  use  a software  cost  estimate 
in  asking  decisions,  we  can  now  examine  the  various  techniques  which  are 
available  for  Baking  software  cost  estimates.  These  techniques  can  be 
categorized  into  five  types:  unit  price,  specific  analogy,  expert  opinion, 

cost  to  cost  relationship,  and  non-cost  to  cost  relationship.  Each  of 
these  techniques  seeks  to  relate  an  historical  coat  to  a future  cost 
(Jones,  1965:19-20).  This  chapter  examines  the  state  of  the  art  of  these 
five  techniques*  The  problems  assoclatod  with  using  them  or  with  devsloplng 
a new  technique  are  a'lso  discussed* 

Unit  Price  Technique 

Probably  the  most  common  estimating  technique  used  to  predict  software 
cost  is  the  unit  price  or  cost  per  instruction  technique.  This  technique 
first  develops  an  expected  cost  for  a single  computer  instruction  in  a 
certain  computer  language.  For  example,  after  examining  previous  JOVIAL 
computer  program  developments,  an  average  or  expected  coat  of  a JOVIAL 
computer  instruction  can  be  determined.  The  second  step  in  the  technique 
is  then  to  size  the  new  software  by  estimating  the  number  of  JOVIAL 
instructions  required.  Multiplying  the  expected  cost  of  a JOVIAL  instruc- 
tion by  the  estimated  number  of  JOVIAL  instruct! yields  the  estimated 
cost  of  the  software  development. 

The  relative  simplicity  of  this  technique  may  -o': in'-,  fox  Its  frequent 
use.  However,  there  are  numerous  problems  associated  with  Ain  technique. 
First,  selecting  an  appropriate  cost  per  instruction  factor  is  difficult 
because  of  the  lack  of  a well  defined  data  base.  Second,  the  technique 
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requires  the  estimator  to  predict  the  number  of  computer  instructions  in  a 
computer  program.  This  is  sometimes  as  difficult  as  predicting  software 
cost.  Third,  there  is  some  question  about  the  accuracy  which  results  when 
the  number  of  instructions  is  used  to  predict  cost.  Since  this  technique 
is  common  in  use,  it  is  important  t<>  understand  each  of  these  problem 
areas, 

"Cost11  Per  Instruction,  While  the  term  "coi»'.'  per  instruction"  might 
appear  simple  and  straightforward,  it  is  anything  but  that.  Defining 
exactly  what  is  meant  by  "cost"  and  what  cost  elements  are  Included  Is 
difficult.  For  example,  does  the  term  cost  only  measure  the  direct  labor 
hours  of  computer  programmers  or  does  it  include  the  direct  labor  of 
managers,  keypunchers,  secretaries,  and  computer  operators?  Is  only  direct 
labor  Included  or  is  the  cost  of  overhead  included?  For  example,  computer 
time  might  represent  as  much  as  2 of  a software  development  cost 
(Wolverton,  197^:629),  However,  some  cost  per  instruction  factors  might 
not  Include  such  a cost. 

In  fact,  any  approach  in  accumulating  software  cost  elements  would 
probably  be  reasonable  as  long  as  the  approach  was  consistent  and  well 
defined.  This  is  a major  problem  since  no  guidelines  exist  which  indlcato 
which  cost  elements  should  be  accumulated  in  determining  a cost  per 
instruction  factor.  Even  if  such  a guideline  existed,  contractors  might 
have  difficulty  in  following  the  guideline  due  to  differences  in  corporate 
accounting  practices.  For  example,  one  company  might  allocate  computer 
costs  based  on  direct  programmer  man-hours  while  another  company  might 
allocate  these  coats  based  on  direct  programmer  salary  dollars.  The  two 
techniques  would  introduce  problems  in  comparing  the  cost  per  instruction 
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factor*  of  two  companies,  Numerous  other  areas  exist  where  difference*  in 
contractors'  accounting  systems  might  preclude  comparison  of  data* 

However,  even  if  these  difficulties  could  be  eliminated,  another 
major  problem  is  encountered  in  measuring  cost  per  instruction.  Software 
development  is  a process,.  The  exact  beginning  and  end  of  the  pieces a are 
not  well  defined.  Some  cost  per  instruction  factors  only  address  the  coot 
during  the  design coding,  and  debugging  phases  of  software  development. 
However,  a software  development  has  a significant  amount  of  activity  prior 
to  and  after  these  phases,  IFor  example,  before  a contractor  c*n  begin  to 
design  a computer  program,  he  frequently  has  to  analyse  tho  user's  opera- 
tional requiremsnts  and  prepare  a system  specification.  System  interfaces 
must  be  defined  and  data  bases  designed.  In  addition,  the  contractor 
frequently  develops  program  test  plans  and  specifies  input/output  message 
formats.  All  of  these  tarhs  consume  resources  during  a software  develop- 
ment prior  to  the  design  of  a single  computer  program. 

After  a programmer  has  debugged  a certain  computer  program,  the  uoft- 
ware  development  is  usually  far  from  complete.  The  contractor  normally 
must  plan  and  conduct  an  integration  test  of  all  programs  and  a system  test 
of  hardware  and  software  components.  Programmers  must  sometimes  train  user 
personnel  or  maintain  the  software  for  a certain  period  after  the  software 
has  been  delivered.  These  costs  are  usually  substantial  and  they  occur 
after  the  software  has  been  debugged. 

If  agreement  can  be  reached  on  what  cost  elements  should  be  measured 
in  the  cost  per  instruction,  then  it  is  equally  important  to  agree  on  what 
parts  of  the  software  process  are  to  be  measured.  Each  part  of  the  pi/ocess 
needs  to  bo  well  defined  and  cannot  simply  be  referred  to  in  broad  terms 
such  as  '•design"  or  "test".  When  both  the  elements  of  cost  and  the  parts 
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of  the  software  development  process  have  been  agreed  upon,  a common  meaning 
for  the  term  "cost"  can  be  established  for  determining  an  appropriate  cost 
per  instruction  factor* 

Cost  Per  "Instruction"*  Unfortunately,  determining  the  appropriate 
cost  would  not  end  the  problems  of  the  term  "cost  per  instruction".  There 
are  considerable  variances  in  the  interpretation  of  tho  term  "instruction6'* 
Por  example,  some  estimators  claim  that  the  instructions  which  should  be 
counted  are  source  instructions*,  They  reason  that  a single  sourco  state- 
ment is  the  product  of  the  programmer  and  is  the  best  measure  of  programmer 
output.  Others,  however,  c aim  that  the  number  of  object  instructions 
should  be  measured.  Object  instructions  are  the  computer  instructions 
which  result  after  the  original  source  instructions  have  been  compiled  by 
the  computer.  This  group  believes  that  tho  use  of  object  instructions  more 
accurately  measures  programming  output. 

While  either  sourco  or  object  instructions  might  bo  appropriate 
measures,  one  cannot  uae  both.  A single  source  instruction  in  JOVIAL  can 
lead  to  five  or  more  object  instructions.  Frequently,  literature  which 
discusses  cost  per  Instruction  factors  does  not  clarify  which  type  of 
Instruction  was  used  in  determining  the  factors. 

Another  problem  exists  in  identifying  tho  language  which  is  being 
used.  Much  of  the  literature  simply  refors  to  the  cost  per  Instruction 
for  an  HOL  (Hlghor  Order  Language)  and  does  not  differentiate  betweon 
Fortran,  Cobol,  or  JOVIAL  developments.  The  use  of  a common  HOL  estimating 
factor  ignores  the  differences  among  languages. 

Still  another  problem  is  whether  or  not  instructions  should  be  classi- 
fied by  type,  A set  of  instructions  in  a time  sensitive  program  might  be 
much  more  difficult  to  develop  than  a set  in  which  time  was  not  a factor. 
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Both  the  System  Development  Corporation  (SDC)  researchers  (Nelson,  196?) 
and  the  Tecolote  researchers  (Frederic,  1974)  found  that  by  classifying 
instructions  into  certain  categories,  the  correlation  between  the  number 
of  instructions  of  a category  and  cost  was  significantly  improved.  Yet, 
despite  the  differences  in  categories  of  instructions,  most  literature 
simply  refers  to  a single  cost  per  instruction  factor  without  identifying 
the  categories  of  instructions  which  led  to  the  particular  cost  factor. 

Perhaps  the  least  obvious  problem  in  counting  the  number  of  instruc- 
tions produced  is  to  determine  which  instructions  should  be  counted.  Mont 
weapon  systs  t,  software  developments  are  not  limited  to  the  development  of 
operational  software  for  the  weapon  system*  Instead  the  development 
includes  assemblers,  '■ompilers,  libraries,  simulators,  test  tools,  and 
data  reduction  programs.  Adding  these  instructions  to  the  instructions 
from  the  operating  programs  yields  the  total  number,  of  deliverable  instruc- 
tions. However,  the  literature  frequently  does  not  identify  whether  the 
cost  per  instruction  factor  was  determined  baaed  on  the  number  of  opera- 
tional computer  instructions  or  the  number  of  delivered  instructions. 

Since  this  difference  can  be  quite  substantial  (Manley,  ^975: 53),  it  is 
imperative  that  the  basis  for  counting  instructions  be  as  well  defined  as 
the  basis  for  defining  cost. 

It  can  be  seen  that  the  simple  phrase  "cost  per  instruction"  Is  not 
quite  that  simple . Any  effort  to  accumulate  a historical  data  base  would 
have  to  ' jquire  that  explicit  definitions  be  established  and  that  accounting 
systems  be  similar.  Without  such  on  effort,  the  data  used  to  develop  a 
cost  per  instruction  factor  would  be  questionable.  This  was  demonstrated 
in  a research  effort  by  Tecolote  Research.  At  the  start  of  their  research, 
they  had  387  data  pointr  from  different  software  development  efforts.  The 
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data  included  cost  information  (sometimes  in  terms  of  man-months)  and  an 
instruction  count.  The  lack  of  information  concerning  the  cost  element#, 
the  software  development  phases  measured,  and  the  typeB  of  instructions 
which  were  counted  led  the  Tecolote  researchers  to  abandon  382  of  their 
387  data  points.  They  then  pro'  -^ded  to  develop  an  estimating  methodology 
based  on  the  five  data  points  about  which  then  had  reasonable  information 
(Frederic,  1974). 

Estimating  The  Number  Of  Instructions.  If  an  appropriate  cost  per 
instruction  factor  has  been  developed  by  careful  data  collection,  the 
estimator  then  needs  to  estimate  the  number  of  instructions  required  in 
the  design  of  a weapon  system.  Weapon  system  software  packages  frequently 
Involve  hundreds  of  thousands  of  instructions  and  an  accurate  estimate  is 
difficult  to  make.  This  difficulty  is  not  unique  to  weapon  system  software 
packages.  For  example,  on  a software  development  by  UNIVAC  for  United  Air 
Lines,  the  initial  estimate  of  the  number  of  instructions  required  for  each 
transaction  was  9,000,  The  system  was  cancelled  when  the  number  of  instruc- 
tions per  transaction  had  escalated  from  9»000  to  146,000  (Schwartz, 
1975:56).  This  type  of  occurence  is  all  too  frequent  in  both  commercial 
and  defense  developments. 

Two  factors  which  affect  the  ability  of  the  estimator  to  predict  the 
number  of  computer  instructions  are  the  experience  of  the  estimator  and 
the  amount  of  detailed  design  information  which  is  available.  To  estimate 
how  many  Instructions  are  required  in  a software  package,  an  estimator 
normally  breaks  the  package  down  into  programs  and  subprograms  until  the 
size  of  the  software  modules  cau  be  estimated  baaed  on  similar  developments 
or  experiences.  Just  as  a junior  electrical  engineer  cannot  architect  the 
hardware  for  an  IBM  360  computer,  neither  can  a junior  programmer  architect 
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or  design  a major  software  syetem*  Breaking  a large  software  system  Into 
small  modules  which  can  be  understood  and  estimated  requires  a high  degree 
of  software  talent*  It  is  the  type  of  talent  found  in  experienced  system 
analysts  or  senior  programmers*  It  requires  a significant  design  effort 
and  cannot  be  done  quickly  un  the  back  of  an  envelope* 

When  breaking  the  software  package  down,  the  estimator  tries  to  get 
down  to  a level  which  is  reasonably  comprehensible*  For  example,  to  say 
that  applications  software  will  be  100,000  instructions  in  size  is  probably 
a guess  and  not  an  engineered  estimate*  However,  to  divide  the  applications 
software  into  programs  and  subprograms  so  that  one  can  say  that  a certain 
message  processing  module  will  require  600  Instructions  Indicates  that  a 
certain  amount  of  engineering  design  and  definition  has  preceded  the 
estimate* 

In  order  to  break  the  software  into  small  modules,  the  software  archi- 
tect or  designer  must  understand  the  functions  and  subfunctions  which  are 
to  be  performod  by  the  system  software.  For  example,  he  must  recognize 
that  a certain  message  processing  function  must  be  performed  by  a software 
module.  This  assumes  that  the  operational  requirements  of  the  user  are 
well  known  and  that  a system  specification  has  been  prepared  which  identi- 
fies software  and  hardware  functions.  However,  the  degree  to  which  such 
design  detail  is  available  to  the  estimator  depends  upon  the  stage  of 
weapon  system  development.  During  the  conceptual  phase,  little  design 
information  is  available  and  an  estimator  must  resort  to  gross  estimates 
such  as  100 , 000  instructions  for  applications  software.  As  the  system 
progresses  through  development,  the  amount  of  design  information  Increases 
to  where  an  estimator  can  now  predict  that  a message  processing  module  1b 
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It  is  important  to  note  that  tho  estimator  requires  both  talont  and 
design  Information,*  Recognizing  that  a message  processing  module  1b 
required  doesnH  help  unless  the  estimator  can  utilize  his  knowledge  or 
experience  to  estimate  the  size  of  the  module.  Likewise,  being  able  to 
size  the  message  processing  module  does  not  mean  that  an  estimator  has 

either  the  capability  or  tho  necessary  design  detail  to  break  a 1Q0S000 

\ 

instruction  fitoftware  package  into  small  600  instruction  modules.  Thus, 
these  two  factors  of  talent  and  design  detail  seriously  impact  tho  ability 
of  the  estimator  to  estimate  the  number  of  instructions. 

Instructions  As  A Predictor  Of  Cost.  If  an  estimator  arrives  at 
reasonable  values  for  the  cost  per  instruction  factor  and  the  number  of 
instructions „ the  resulting  cost  estimate  still  might  not  be  reliable. 

The  problem  is  that  the  validity  of  using  the  number  of  instructions  as  a 
predictor  of  cost  is  questionable.  First,  the  use  of  a coat  per  instruc- 
tion factor  does  not  specifically  address  other  factors  which  Influence 
software  cost.  A study  by  the  System  Development  Corporation  (SDC)  for 
the  Electronic  Systems  Division  of  the  Air  Force  Systems  Command  identified 
94  variables  which  Influence  software  cost.  These  variables  address  not 
only  the  properties  of  the  software  and  the  weapon  system,  but  also  address 
external  factors  which  can  affect  the  software  acquisition  process.  Using 
a simple  coat  per  instruction  factor  ignores  many  other  variables  which 
can  Impact  cost. 

Second,  the  relationship  between  the  number  of  instructions  and  cost 
has  not  been  well  defined  by  past  research  efforts.  Much  of  the  literature 
simply  assumes  that  the  relationship  is  linear  and  applies  the  same  cost  per 
Instruction  factor  to  developments  which  are  1,000  or  1,000,000  instructions 
in  size.  However,  a few  research  efforts  postulate  exponential  relation- 
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ships  between  cost  and  instructions. 

The  SDC  study  (Farr  and  Nanus,  1964:242)  developed  the  following 
relationship: 

1 5 

Man-months  of  Effort  = (Constant)  x (Number  of  Instructions) 

On  the  other  hand,  Tecolote  Research  (Frederic,  1974:34)  using  probably 
the  largest  number  of  data  points  (i.e.  387)  found  tho  data  beet  described 

toy: 

Cost  (in  FY73  • K)  ■ 0,079  (Number  of  Instructions)0*8^ 

(Again,  however,  the  Tecolote  researchers  found  such  a low  correlation 
between  this  relationship  and  the  data,  that  they  eventually  abandoned 
almost  all  of  their  387  data  points  and  built  a cost  model  around  only 
five  points.)  Both  the  effort  by  Farr  and  Nanus  and  the  effort  by 
Tecolote  utilized  the  number  of  delivered  object  instructions  and  yet  the 
results  differed  significantly, 

A third  research  effort  by  IBM  (Malone,  1975:1-8)  baaed  its  study  on 
the  number  of  source  statements.  Their  findings  yielded  the  following 
relationship: 

Man-months  = .00007  (Number  of  Instructions)1*1^8^* 

Again,  this  result  1b  significantly  different  from  the  other  two  research 
efforts. 

Each  of  the  three  .research  efforts  did  not  recommend  the  use  of  their 
findings  to  estimate  software  cost.  This  was  due  to  the  problems  of 
gathering  comparable  data,  to  the  small  data  samples,  and  to  the  low 
correlation  coefficients  which  were  obtained. 

It  is  clear  that  the  relationship  between  the  number  of  Instructions 
and  coat  is  uncertain.  Whether  the  relationship  is  linear,  or  exponential 
and  what  values  of  constants  should  be  used  are  currently  uncertain.  Not 
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until  a detailed  and  extensive  research  effort  has  verified  the  predictive 
relationship  between  the  number  of  instruction*  and  cost  will  un  estimator 
be  able  to  apply  this  estimating  technique  with  a good  degree  of  confidence. 

Pae  Of  Coat  Per  Instruction  Technique.  Despite  the  numerous  problems 
associated  with  its  use,  the  cost  per  instruction  technique  is  and  probably 
will  remain  the  most  utilized  technique.  Its  prevalence  is  perhaps  duo  to 
its  simplicity  and  the  appearance  it  provides  of  a scientific  approach. 

It  implies  that  an  appropriate  cost  per  instruction  factor  has  been  selected 
based  on  similar  software  developments.  It  also  implies  that  a detailed 
software  design  has  been  performed  to  determine  the  number  of  instructions 
in  the  software  package.  Whether  or  not  these  two  implications  are  in  fact 
true  should  be  strongly  questioned  by  managers,  engineers,  and  cost 
estimators  who  are  presented  with  cost  estimates  based  on  this  technique. 

Specific  Analogy 

A second  coat  estimating  technique  is  specific  analogy  where  future 
software*  costs  are  estimated  from  the  hintorical  costa  of  a similar  software 
development.  This  technique  is  frequently  used  in  estimating  the  cost  of 
manufactured  goods.  For  example,  if  previous  production  costB  for  a tele- 
vision set  are  known,  a managor  can  use  tills  information  to  predict  future 
production  costs  of  a similar  television  set. 

However,  in  applying  this  technique  to  software,  there  are  two  major 
problems.  First,  there  is  limited  historical  data  concerning  the  software 
cost  of  past  weapon  system  developments  (Air  Forco  OSR,  1973:46).  Second, 
the  nature  of  software  is  such  that  there  is  usually  limited  similai’lty 
between  any  two  software  developments. 

Historical  Data.  To  utilize  the  specific  analogy  technique,  an 
estimator  must  first  determino  the  software  costa  associated  with  a similar 
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program.  This  is  difficult  for  a number  of  reasons.  Until  recently, 
software  coots  wore  generally  not  separated  from  the  hardware  cost  of  a 
weapon  system.  Work  breakdown  structures  which  encourage  detailed  software 
cost  and  status  reporting  are  still  lacking  (Ashe  et  al,  1975: Vol  2,  2-35). 

A few  programs  have  required  that  software  cost  be  separately  identi- 
fied and  recorded.  Unfortunately,  that  data  suffers  from  tho  lack  of 
definition  which  has  previously  been  discussed.  There  is  no  common 
agreement  or  standard  which  details  what  elements  of  cost  are  software 
related.  For  example,  during  a system  test  of  hardware  and  softwaro,  ; 

programmero  are  required  to  develop  test  plans  and  to  modify  programs. 

Whether  those  costa  are  charged  to  system  test  or  to  software  dependB  upon  ,i 

the  accounting  system  and  the  work  breakdown  structure.  Again,  the  cost 
elements  and  the  phases  of  software  development  which  are  included  in 
available  coot  data  are  not  well  defined. 

Until  a“ common  set  of  definitions  and  a common  data  collection  require-  ’ 

ment  is  levied  upon  major  weapon  system  developments,  a software  cost 
estimator  using  the  specific  analogy  technique  will  probably  have  to  rely 

on  the  use  of  only  one  or  two  historical  data  points  gathered  by  personal  j 

\ 

contact  or  personal  experience.  j 

Software  Similarity,  After  determining  the  software  cost  of  a similar  | 

development.,  the  estimator  must  then  make  a somewhat  subjective  set  of 

A 

Judgments  concerning  the  similarity  of  the  new  development  with  the  old  j 

development.  It  is  very  unlikely  that  the  new  system  will  utilize  the  same  j 

hardware  or  be  required  to  perform  the  same  functions.  Therefore,  drawing  j 

parallels  between  systems  is  difficult.  For  example,  cost  estimators  used 


the  software  cost  of  the  Air  Force  Back  Up  Interceptor  Control  (BUIC)  to 
estimate  the  software  cost  of  the  Air  Force  SAGE  System  (Jones,  ly65:19). 
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While  somewhat  similar  in  mission,  the  two  systems  are  quit*  different  and 
significant  subjective  Judgment  vas  required  to  assess  these  differences 
and  to  assign  a cost  to  them. 

Other  systems  such  as  the  Airborne  Warning  and  Control  System  (AWACS) 
have  few  similar  development*  upon  which  to  draw  comparisons.  Advancing 
technology  continues  to  introduce  n*w  hardware  and  software  techniques 
which  compound  the  problem  of  finding  a similar  program  to  use  for  an 
Analogy,  Again,  oven  if  a similar  program  is  identified,  the  problem  of 
obtaining  accurate  historical  cost  data  remains. 

Use  Of  The  Specific  Analogy  Technique.  Despite  these  problems,  the 
specific  analogy  technique  is  still  in  active  use.  While  it  is  subjective 
in  its  application,  it  does  take  into  account  the  real  world  problems  and 
performance  which  actually  occurred  on  a similar  program.  For  example,  if 
an  estimator  used  the  BUXC  cost  data  to  estimate  the  SAQE  system,  the 
estimator  will  automatically  Include  those  costs  which  were  associated  with 
funding  difficulties,  changing  requirements  or  other  real  world  problems. 

The  specific  analogy  technique  forms  the  primary  basis  of  a formal 
Army  softwara  cost  estimating  teohnique.  An  Army  catalog  provides  detailed 
cost  information  for  20  data  systems.  An  estimator  can  broaden  his  know- 
ledge by  utilizing  the  recorded  experiences  found  in  the  catalog.  Adjust- 
ments in  the  historical  cost  indications  duo  to  differences  in  the  new 
system  are  then  identified  and  explained  (Department  of  Army,  1975), 

While  subjective  in  nature,  the  specific  analogy  method  tends  to  mini- 
mize optimistic  schedules  and  estimates  since  it  is  based  on  actual  perform- 
ance data  for  a similar  system.  With  better  collection  and  recording  of 
historical  software  cost  data,  this  technique  holds  such  promise,  A 
research  effort  is  currently  underway  by  the  Rome  Air  Development  Center 
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of  the  Air  Force  Systems  Compand  to  develop  the  historical  data  base  which 
la  required  for  the  use  of  this  technique  (Nelson  and  Sukert,  1974)* 

One  frequent  criticism  of  this  technique  is  that  it  is  not  very  objec- 
tive or  scientific*  While  an  estimator  who  states  that  the  applications 
scftware  consists  of  100,000  instructions  might  be  thought  to  have  a 
scientific  basis,  an  estimator  who  states  that  a system  will  require  50% 
more  for  software  is  felt  to  be  subjective  and  unscientific.  In  fact, 
neither  technique  by  itself  has  any  inherent  scientific  credibility  or 
objectivity.  Instead,  it  is  the  conscientious  and  documented  application 
of  these  techniques  which  can  make  either  one  a reasonable  tool  for  pro- 
ducing a reliable  cost  estimate. 

Again,  the  lack  of  historical  data  limits  the  specific  analogy 
technique.  Current  and  future  data  collection  efforts  may  eliminate  this 
problem.  The  remaining  difficulty  will  be  in  the  subjective  extrapolation 
0;i  the  cost  of  one  program  to  another  "similar1'  program, 

Expert  Opinion 

A third  software  technique  for  estimating  software  cost  is  the  use  of 
expert  opinion.  The  title  is  self  explanatory.  To  estimate  software  cost 
one  simply  asks  an  expert  or  a group  of  experts  to  use  their  knowledge 
and  experience  in  predicting  the  cost  of  software.  Two  methods  for 
obtaining  an  expert  opinion  are  the  engineering  cost  analysis  and  the 
Delphi  method. 

Engineering  Cost  Analysis.  Despite  its  somewhat  authoritarian  title, 
an  engineering  cost  analysis  is  simply  an  expert  opinion.  A software 
export  or  software  engineer  is  presented  with  a functional  description  of 
the  weapon  system.  He  then  proceeds  to  analyze  the  software  usually  by 
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breaking  the  software  Into  smaller  programs  and  subprograms.  When  the 
software  haB  been  divided  into  small  and  comprehensible  modules,  the 
expert  then  estimates  the  amount  of  resources  required  for  each  module 
using  his  knowledge  and  experience  as  a guide.  For  example,  the  expert 
might  identify  a message  processing  module  which  in  his  opinion  will 
require  three  months  of  effort  by  a junior  programmer  and  six  hours  of 
central  processor  time,  A cost  analyst  then  transforms  the  estimated 
man-months  and  computer  time  into  a dollar  cost. 

Some  problems  with  this  technique  have  previously  been  discussed. 

The  technique  requires  an  ’'expert"  which  is  a vague  term  referring  to 
someone  who  can  predict  software  resources  accurately  based  upon  his 
education  and  experience.  If  experts  were  identified  based  on  demonstrated 
estimating  accuracy,  the  technique  might  be  ideal.  However,  ouch  identifi- 
cation is  not  common  in  practice  and  true  "experts"  are  difficult  to  find. 

Also,  an  engineering  coat  analysis  requires  that  the  functional  design 
of  the  weapon  eystem  be  fairly  complete.  Decisions  concerning  which 
functions  are  to  be  performed  by  hardware  or  by  software  have  to  be  made 
before  an  engineering  cost  analysis  of  the  software  can  be  made.  Explicit 
and  detailed  documentation  of  the  user's  requirements  must  be  available. 

Use  Of  Engineering  Cost  Analysis.  Despite  these  problems,  the  engineer- 
ing coat  analysis  has  some  unique  strengths.  It  requires  that  an  engineer 
sit  down  and  design  the  software  into  small  enough  modules  so  that  he  can 
understand  the  resources  required  to  develop  the  module.  If  the  functional 
modules  aro  fairly  small  (e.g.  three  man-months),  then  the  program  manager 
can  he  reasonably  certain  that  a fairly  detailed  design  has  been  made  and 
that  the  user's  requirements  were  known  to  sufficient  certainty  to  support 
the  detailed  design.  On  the  other  hand,  if  the  estimate  makes  frequent 
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references  to  large  modules  (e.g,  five  raan-years),  than  the  program 
manager  Bight  infer  that  either  a detailed  design  has  not  been  performed 
or  that  the  user's  requirements  were  uncertain  and  forced  the  engineer  to 
group  requireaents  into  large  modules. 

Thus,  the  use  of  an  engineering  coot  analysis  cam  provide  the  program 
manager  with  am  insight  into  the  degree  with  which  user  requirements  are 
known.  Since  uncertain  user  requirements  are  a major  cause  of  system  over- 
runs, the  use  of  an  engineering  coBt  estimate  may  provide  the  manager  with 
valuable  information. 

Another  benefit  from  the  use  of  an  engineering  cost  amalysls  ±u  that 
it  takes  into  account  tha  unique  nature  of  the  new  program.  The  impact 
of  certain  interfaces  or  timing  requirements  can  be  individually  addressed 
instead  of  basing  their  costs  on  the  average  historical  costa  of  previous 
systems.  For  this  reason  and  for  others,  the  use  of  an  engineering  cost 
analyeis  is  recommended  by  most  of  the  researchers  into  the  area  of  soft- 
ware cost  estimating. 

The  Delphi  Method.  A second  method  for  obtaining  a cost  estimate  from 
a group  of  experts  is  the  Delphi  method.  Developed  by  the  HAND  Corporation, 
Delphi  first  gathers  estimates  from  individual  experts.  The  results  of  the 
individual  estimates  are  then  iteratively  fed  back  to  the  experts  until  s 
consensus  is  reached.  The  Delphi  method  is  not  a committee.  The  experts 
do  not  meet  face  to  face.  Instead,  the  experts  are  given  the  results  of 
each  iteration  and  asked  to  explain  or  justify  their  opinions.  These 
opinions  and  reasons  are  then  given  to  the  other  experts  until  eventually 
the  estimates  of  the  experts  converge. 

RAND  performed  an  experiment  using  Delphi  to  estimate  a particular 
project's  software  cost.  Two  teams  of  experts  were  used  and  guided  by  the 
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Delphi  method.  Their  estimates  were  217  man-months  and  1090  mam-month* 
respectively.  The  reasons  for  the  wide  difference  were  not  clear.  Two 
additional,  groups  of  experts  were  also  asked  to  estimate  th*  ease  effort 
using  a simple  committee  technique.  Their  results  were  485  man-months  and 
656  main-months  respectively.  The  actual  cost  ifas  4-89  main-months  which 
would  tend  to  indicate  that  tho  use  of  the  committee  technique  might  ho 
better  than  the  Delphi  method  (Farquhar,  1970). 

Delphi  Versus  Committee.  Tho  purpose  of  presenting  the  Delphi  method 
ie  not  to  approve  or  disapprove  its  use.  The  purpose  is  to  examine  why  a 
face  to  face  committee  of  experts  appears  to  perform  better  than  a faceless 
group  of  experts.  In  a study  by  another  RAND  researcher,  the  Delphi  tech- 
nique was  found  to  be  am  unreliable  method  for  estimating.  He  found 
considerable  evidence  that  there  was  no  difference  between  the  estimates 
of  laymen  and  experts  and  suggested  that  Delphi  loads  to  a manipulated 
group  suggestion  and  not  a true  consensus  (Sackman,  1975).  Perhaps  the 
reason  that  the  committee  outperformed  the  Delphi  method  was  that  it 
encouraged  face  to  face  confrontation  which  enabled  the  group  to  judge 
whether  someone  was  a true  "expert*  and  to  reach  a real  consensus. 

The  point  is  that  when  obtaining  estimates  from  a group  of  experts,  it 
appears  advisable  to  have  th*  experts  engage  face  to  face.  Iterative  formal 
communication  between  different  agencies  might  result  with  estimates 
similar  to  the  Delphi  method.  On  tho  other  hand,  the  .iso  of  a committee 
to  examine  and  explore  the  differences  in  cost  estimates  might  result  in 
improved  communications  and  better  estimates. 

Coat  To  Cost  Estimating  Technique 

A fourth  technique  for  estimating  softwar#  cost  uses  the  cost  of  one 
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part  of  the  weapon  system  to  estimate  the  cost  of  another  part  of  the 
system.  This  technique  is  frequently  used  to  estimate  the  cost  of  system 
test  or  of  initial  spares.  Both  of  these  costs  can  be  estimated  baaed 
upon  an  historical  percentage  of  the  prime  mission  equipment.  For  example, 
initial  spares  for  similar  ground  electronic  systems  might  be  an  average 
20%  of  prime  mission  equipment  cost.  Once  a djtailod  estimate  of  the 
primj  mission  equipment  for  the  new  system  has  been  made,  the  estimator 
can  then  simply  apply  the  20%  factor  to  that  cost  to  estimate  the  cost 
of  initial  spares.  This  technique  works  quite  well  with  a limited 
number  of  items  (Jones,  1965:26). 

However,  applying  this  technique  to  software  is  difficult.  First, 
there  again  is  the  problem  of  insufficient  historical  cost  information. 
Second,  software  constitutes  almost  90%  of  some  systems  such  as  WWMCCS 
and  only  2%  of  the  B-l  (Ashe  et  al,  1975:2-46).  Using  an  average  cost 
factor  for  software  simply  does  not  work  as  well  as  it  does  for  spares  or 
other  system  components.  Despite  these  problems  there  are  ways  in  which 
the  cost  to  cost  technique  can  be  used  in  a limited  fashion. 


r 


For  example,  designing,  coding,  and  testing  a program  might  be 
expected  to  average  40%,  20%  and  40%  respectively  of  total  computer  program 
development  costs.  Once  the  design  of  a particular  program  has  been 
completed,  a manager  can  estimate  the  expected  coats  for  the  remaining 
activities  of  coding  and  testing.  As  a rough  rule  of  thumb,  this  tech- 
nique has  some  merit.  However,  there  are  significant  differences  in  the 
percentages  of  total  cost  attributed  to  any  one  activity  (Boehm,  1975:7). 
Whether  these  differences  occur  becauBO  of  the  lack  of  distinctness  in  the 
definition  of  the  activities  or  whether  these  differences  are  attributable 
to  some  other  cause  is  uncertain.  Again,  limited  coBt  data  and  poor 
definition  limit  the  accuracy  and  utility  of  this  technique. 
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Hon-Coat  To  Cost  Relationship  Technique 

The  fifth  category  of  software  coat  estimating  technique  la  the  uae  of 
a non-coat  parameter  to  estimate  coat.  Frequently  called  parametric  esti- 
mating, it  la  by  far  the  moat  aophlatlcated  and  ambitious  of  all  the 
techniques.  The  technique  develops  parametric  equations  where  character- 
iatica  or  parameters  of  a now  computer  program  are  uaod  to  ostlmato  itu 
coat.  These  characteristics  Include  auch  parameters  am  the  numbor  of 
source  instructions,  the  type  of  weapon  system,  the  age  of  the  computer 
upon  which  the  program  will  run,  the  type  of  compiler,  and  numerous  other 
characteriatica.  One  parametric  research  effort  by  the  System  Development 
Corporation  (SDC)  identified  over  90  parameters  which  influence  cost 
(Nelson,  1967),, 

Once  these  parameters  are  identified,  the  cost  estimating  researcher 
attempts  to  colloct  a large  sample  of  data  ao  that  the  relationship  between 
theae  non-coat  parameters  and  software  coat  can  be  established  using 
regression  techniques.  In  order  to  havo  a statistically  valid  parametric 
equation,  roaearchera  attempt  to  obtain  as  many  data  points  as  possible. 
This  search  for  many  data  la  the  source  of  one  of  the  problems  with  this 
technique. 

For  example,  a researcher  might  have  data  on  five  JOVIAL  developments, 
five  Fortran  developments  and  flvo  Cobol  developments.  Rather  than  develop 
threo  models  based  on  only  five  data  points  each,  a researcher  is  likely 
to  lump  all  of  the  fifteen  points  together  and  develop  a single  model  for 
use  on  programs  written  in  either  JOVIAL,  Fortran,  or  Cobol,  While  the  use 
of  fifteen  points  appoara  to  give  the  model  a greater  statistical  basis, 
an  implicit  assumption  is  that  tbsro  ie  no  cost  difference  due  to  tho  use 
of  different  computer  languages.  Further  lumping  of  data  as  to  the  type 
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of  computer,  the  particular  compiler  and  other  factor*  la  made  until  the 
parameters  used  in  the  equations  are  quite  broad* 

This  lumping  of  data  may  be  regrettable,  however,  it  is  the  only 
possible  coarse  for  researchers  who  are  faced  with  a scarcity  of  data*  Xa 
fact,  the  lumping  of  data  ha*  been  highly  *ucc#*sful  in  other  'reas*  In 
estimating  the  production  coat  of  an  airframe,  knowlodge  of  only  three 
parameters  (e,g,  weight,  speed,  and  production  quantity)  are  sufficient  to 
reasonably  predict  the  production  cost  of  an  aircraft  (Levenson  et  al,  1971)* 

Software  oost,  however,  appear*  to  be  dependent  on  more  variables  than 
other  product**  A technique  using  only  two  or  three  parameters  suffer*  in 
predictive  pow*r.  However,  a technique  which  requires  the  determination 
of  many  factor*  is  unlikely  to  be  popular  in  u*e,  especially  if  the 
determination  of  a proper  value  for  the  various  factors  is  difficult. 

Thus,  researchers  not  only  have  a problem  in  finding  appropriate  parametric 
equations,  they  also  must  attempt  to  minimize  the  number  of  Inputs  inquired 
for  the  uso  of  the  equation. 

Researchers  have  frequently  developed  relationships  among  parameters 
whioh  are  quite  different  from  the  efforts  of  other  researcher*.  This  is 
perhaps  duo  to  the  lack  of  clear  definitions  in  the  area  and  to  the  small 
data  samples  available  to  the  researchers.  However,  the  impact  is  that  uo 
single  parametric  technique  has  received  a large  degree  of  acceptance  or  p 
wide  dogreo  of  use.  To  understand  the  problems  with  developing  or  utilizing 
parametric  techniques  we  con  examine  five  of  them.  Throe  of  them  are  well 
publicized  and  in  limited  use.  These  three  techniques  are  those  by  the 
Syntom  Development  Corporation,  by  J,D.  Aron,  and  by  Ray  Wolverton,  A 
fourth  technique  is  a recontly  developed  technique  used  by  the  Computer 
Systems  Command  of  the  U,S,  Army  while  the  last  one  is  one  developed  by 
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Tecolote  Research  for  the  Navy* 

SDC  Tecfy^gu*.  The  SDC  technique  was  developed  under  contract  to  BSD 
SroK  .'964  to  1968.  The  results  of  this  massive  research  effoid;  are  con- 
tained in  nine  volumes  identified  in  the  bibliography.  The  Handbook 
written  by  E.A,  Nelson  in  1967  summarized  most  of  the  research  findings. 

To  gather  data,  the  SDC  researchers  resorted  to  the  use  of  question- 
naires, While  this  limited  the  accuracy  of  the  data,  they  did  amass  169 
data  points  which  remains  the  largest  software  cost  data  base  currently 
published  by  any  single  group  of  researchers  (Morin,  1974),  The  SDC 
researchers  identified  over  90  variables  and  examined  their  Influence  upon 
cost  during  different  phases  of  ths  software  dsvelopmsnt  process. 

For  the  design,  code,  and  test  phase  of  software  development,  the 
researchers  were  able  to  develop  a parametric  equation  based  on  fourteen 
different  parameters.  Again,  while  the  researchers  examined  over  90 
variables,  their  final  equation  la  based  only  on  14,  This  is  because  the 
remaining  variables  do  not  improve  the  predictive  power  of  the  parametric 
equation.  The  equation  is  worthwhile  to  examine  in  depth.  It  is: 

Y(l)  = - 33.63  + 9.15  X(3)  + 10.73  X(8)  + .31  X(26)  + .46  X(30) 

+ .40  X(41)  + 7.28  X(b2)  - 21,45  X( 48.1)  + 13.53  X(48.5) 

+ 12.35  X( 51)  + 58.82  X(53)  + 30.61  X(56>  + 29.55  X(72) 


+ .54  X(75)  - 25.2  X(?6) 


Where  the  variables  have  the  following  interpretations: 

Y(l)  - Total  man-aonths  required 

X(3)  - Lack  of  knowledge  of  operational  requirements 

X(8)  - Stability  of  design 

X(26)  - Percent  mathematical  instructions 


X(30) 


Percent  information  and  storage 
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X(41)  - Number  of  subprograms 

X(48.1)  - Business 

X(48.5)  - Stand  alone 

X(51)  - First  program  on  comp?  ^r 

X(53)  - ADP  components  developed  concurrently 

X(56)  - Random  access  device  used 

X(72)  - Different  computers  for  programming  and  operation 

X(75)  - Number  of  man  trips 

X(76)  - Program  data  point  developed  by  military  organization 

The  purpose  of  reproducing  this  equation  here  is  not  to  recommend  its 
use.  Instead,  one  should  note  the  conspicuous  absence  of  one  variable 
from  the  parametric  equation.  The  SDC  equation  for  predicting  man-months 
does  not  utilize  the  number  of  computer  instructions  as  an  independent 
variable.  None  of  the  other  variables  have  a very  direct  relationship  with 
the  number  of  instructions.  The  lack  of  this  variable  implies  that  the  SDC 
researchers  found  little  improvement,  if  any,  in  their  parametric  equation 
when  adding  the  number  of  instructions  to  the  input.  This  is  important  to 
note  since  most  other  parametric  estimates  depend  heavily  (if  not  exclusively) 
on  the  number  of  instructions  as  an  input  (Morin,  1974) * 

Recognizing  the  importance  of  the  number  of  instructions  in  other 
research  efforts,  the  SDC  handbook  provides  tables  which  Indicate  the 
number  of  man-months  of  effort  required  to  design,  code,  and  test  1000 
instructions  of  a particular  language.  If  we  look  at  some  entries  in 
those  tables,  we  might  better  understand  the  conspicuous  absence  of  the 
number  of  instructions  from  the  SDC  parametric  equation.  Two  of  the 
entries  are: 
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JOVIAL  (1000  object  instructions) 

..  ...  Man-months  of  effort  required 

Ho.  of  data  — 

points*  Max  Min  Std  Dev  Median  Mean 

15  7.6  .66  2.31  2.5  3.07 

JOVIAL  (1000  aource  instructions) 

„ ...  Man-months  of  effort  required 

No.  of  data  — — — ---■  1 -*■ 1 — ■ — 

points*  Max  Min  Std  Lev  Median  Mean 

15  ' 46.25  2.13  12.01  6.15  10.2? 

♦(Note:  Each  data  point  represented  a separate  software 

development  project). 

Thus,  the  san-months  of  effort  required  to  design,  cods,  and  test 
1000  JOVIAL  source  Instructions  was  only  2.13  Ban-months  on  one  software 
project,  while  it  took  46.25  man-mor.ths  to  design,  code,  and  teat  1000 
JOVIAL  instructions  on  another  program.  The  large  standard  deviation 
Indicates  that  there  was  considerable  variability  in  the  data  and  it  is 
easy  to  understand  why  the  SDC  researchers  did  not  inculde  the  number  of 
instructions  as  a parameter  in  their  equation. 

Unfortunately,  by  publishing  a mean  or  expected  number  of  man-months 
for  1000  JOVIAL  instructions,  the  researchers  provided  information  which 
a novice  cost  estimator  might  take  out  of  contsxt.  The  variance  in  the 
data  indlcatea  that  the  number  of  instructions  is  either  a poor  estimator 
of  man-months  or  that  the  SDC  data  was  faulty.  Under  either  circumstance, 
use  of  tha  mean  from  this  table  would  be  of  questionable  value.  This  is 
pointed  out  because  in  Chapter  5 we  shall  see  that  some  estimators  do  use 
this  mean. 

While  the  SDC  data  may  be  questioned,  it  represents  a major  research 
effort  in  an  area  where  few  true  research  efforts  have  been  made.  It  still 
repressnts  probably  the  largest  software  cost  data  base  ever  assembled  by  a 
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.research  team*  The  large  number  of  variables  examined  as  well  as  the 
researchers'  comments  concerning  the  impact  of  these  variables  upon  cost 
Is  interesting  and  valuable*  Unfortunately,  the  SDC  research  was  not 
continued  until  a large  enough  data  base  could  be  developed  from  controlled 
data  to  produce  parametric  equations  with  reasonable  confidence  intervals 
(Nelson,  1967), 

Aron's  Technique*  Published  in  1970,  Aron's  techniquo  Is  frequently 
cited  in  the  literature.  His  paramstrlc  methodology  not  only  identifies 
the  variables,  but  also  includes  recommended  values  for  these  variables 
based  on  a large  number  of  major  104  system  programs.  While  Aron  does  not 
publish  the  data  upon  which  his  techniquo  is  based,  his  refsrencs  to  a large 
number  and  variety  of  IBM  programs  adds  a certain  credibility  to  his  article. 

The  Aron  technique  begins  with  a system  design  and  an  estimate  of  the 
number  of  deliverable,  assembly  level  instructions.  The  difficulty  of 
these  Instructions  are  than  judged  to  be  easy,  medium,  or  ha*d  primarily 
based  upon  the  nuraoer  of  interactions  a particular  program  will  have. 

Nest,  the  estimator  utilizes  the  data  reproduced  in  Table  1.  The  table 
indicates,  for  example,  that  for  a hard  progr.*jn  being  performed  in 
12-24  months,  one  can  expect  a productivity  of  125  assembly  level  instruc- 
tions per  man-month.  After  applying  the  appropriate  productivity  factors 
to  each  program,  tho  result  is  multiplied  by  a factor  of  2.5  to  account 
for  management  and  support  personnel. 
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Row  1 


Row  2 


Row  3 


Duration 

Difficulty^. 

6-12 

Months 

12-24 

Months 

More  Than 
24  Months 

Easy 

20 

500 

10,000 

M odium 

10 

250 

5,000 

Hard 

5 

125 

1,500 

Instructions 

Instructions 

Instructions 

Units 

per 

per 

per 

Man-Day 

Man-Month 

Man-Year 

Very  Few 
Interaction* 

Some 

Interacfiono 

Many 

Interactions 


<*& 


Table  I 

Productivity  Table 

This  is  an  extremely  brief  simplification  of  Aron’s  12  page  article. 
The  primary  point  ia  that  Aron’s  technique  i a heavily  dependent  on  the 
number  of  instructions.  Also,  two  things  can  be  noted  from  Table  1, 

First,  there  ia  a large  range  of  programmer  productivity  based  on  a 
somewhat  subjective  evaluation  of  the  degree  of  difficulty  associated  with 
a program.  Second,  Aron  states  that  if  a project  extends  over  two  years, 
then  programmer  productivity  Increases  due  to  a learning  effect.  Aram, 
offers  no  proof  of  a learning  effect,  and  no  other  available  literature 
supports  this.  However,  talcing  Aron’s  data  at  face  value,  one  can  see 
that  programmer  productivity  goes  from  a low  of  20  instructions  per  man- 
day  for  an  easy  program  on  a short  schedule  to  a high  of  33  Instructions 
per  man-day  (10,000/year)  for  an  easy  program  on  a longer  schedule  (Aron, 
1970). 
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Wolverton  Technique.  A third  technique  was  published  by  Hay  Wolverton 
In  197^  based  on  experience  at  TRW.  The  21  page  article  describes  the 
technique  In  sons  detail  but  no  specific  mention  is  made  of  the  data  base 
used  in  developing  the  technique.  Also,  no  Indication  Is  given  as  to  how 
well  the  estimating  technique  fits  the  data  base. 

The  technique  begins  with  a design  in  which  software  modules  are 
identified  and  then  categorised.  The  categories  are  old  and  new  for  the 
first  split  and  then  easy,  medium,  and  difficult  for  another  split.  This 
yields  six  categories  of  difficulty.  The  modules  are  then  further  separated 
according  to  the  type  of  function  being  performed.  Six  functions  such  as 
control  and  input/output  are  identified.  This  further  divides  the  software 
into  36  different  categories  based  on  newness,  difficulty  and  type  of 
program. 

Wolverton  then  provides  a table  of  cost  per  instruction  factors  for 
each  of  the  36  categories.  The  number  of  object  instructions  i.i  each  of 
the  36  categories  is  then  multiplied  by  the  appropriate  cost  per  instruc- 
tion factor  to  yield  a total  cost  including  overhead.  The  cost  per  instruc- 
tion factors  range  from  $15  to  $75  depending  on  the  category. 

A primary  strength  of  the  Wolverton  technique  is  its  simplicity.  It 
requires  some  fairly  simple  subjective  judgments  to  place  the  software 
into  36  categories.  Similar  to  the  Aron  technique,  it  only  uses  a few 
characteristics  of  the  software  which  are  reasonably  well  known  with  the 
exception  of  the  number  of  instructions.  In  fact,  the  major  problem®  with 
the  technique  are  that  it  requires  an  estimate  of  the  number  of  object 
instructions  and  that  it  assumes  a linear  relationship  between  cost  and 
the  number  of  instructions  within  each  of  the  36  categories. 

While  the  technique  is  attractive  because  of  its  simplicity,  its  use 
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is  questionable  since  no  indication  is  provided  as  to  the  nature  of  the 
original  data.  How  well  the  technique  explains  the  original  data  should 
be  known  before  one  can  use  a technique  with  any  confidence  (Wolverton, 
1974). 

AOPREP  Technique,  A fourth  technique  has  been  developed  for  the  U.S. 
Army  Computer  Systeaa  Command  by  the  Planning  Research  Corporation  (PRC). 

The  technique  is  called  Automatic  Data  Processing  Resource  Estimating 
Procedures  (ADPREP).  The  technique  is  based  on  data  collected  from  20  Army 
data  automation  systems.  Half  of  these  systems  were  new  while  the  remainder 
were  major  revisions  of  existing  systems.  The  systems  were  business  type 
systems  and  required  a level  of  effort  averaging  105  man-months. 

The  technique  provides  a number  of  estimating  guidelines.  For  example , 
detailed  estimating  worksheets  addressing  each  phase  of  development  are 
provided.  An  estimator  can  review  the  historical  values  of  the  20  programs 
and  determine  an  appropriate  worksheet  value  based  on  analogy  or  judgment, 
ICumerous  heuristics  for  certain  specific  activities  such  as  documentation 
are  also  provided. 

However,  an  assential  part  of  ADPREP  are  the  parametric  equations 
which  were  developed  using  the  data  from  the  20  Army  systems.  We  will 
review  one  of  those  equations  which  addresses  the  ease  variable  us  the  SDC 
equation.  The  particular  equation  was  developed  based  on  the  data  from 
10  new  Army  systems  and  predicts  the  total  man-months  of  effort  to  design, 
code,  and  test  the  software.  The  equation  is: 

Y(2)  = 2.57  x<2)  + 5.10  x(3)  + 0.12  X<5) 

Where  the  variables  have  the  following  interpretation: 

Y(2)  - Personnel  requirements  in  man-months 
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X(2)  - Total  number  of  different  output  formats  of  AJDPS  products 

X(3)  - Total  number  of  record  types  in  data  base 

X( 5)  - Average  number  of  transactions  per  month  of  input  in 

thousands 

Note  that  in  contrast  to  the  14  variables  used  by  the  SDC  equation, 
the  ADPREP  requires  only  three  variables  to  predict  man-months.  Also,  like 
the  SDC  equation,  the  ADPREP  equation  dooa  not  rely  on  the  number  of 
instructions  in  developing  its  estimate.  Again,  while  some  techniques  are 
strongly  baaed  on  the  number  of  computer  instructions,  other  parametric 
techniques  do  not  utilize  it  at  all. 

The  degree  to  which  the  estimating  equation  explains  the  sample  data 
Is  also  provided  in  the  ADPREP  documentation.  The  ADPREP  manual  claim* 
that  the  above  estimating  equation  has  a squared  multiple  correlation 

p 

coefficient  (R~)  of  1.0.  Such  a high  correlation  coefficient  makes  the 
ADPREP  technique  very  suspect.  In  essence,  the  ADPREP  claim  is  that  the 
estimating  equation  explains  all  of  the  variance  found  in  the  sample  data 
and  that  all  ten  random  data  points  were  exactly  on  the  line  defined  by  the 
estimating  equation.  An  exact  fit  is  quite  questionable  considering  the 
random  nature  of  the  variables  concerned. 

Nevertheless,  the  ADPREP  technique  is  quite  well  presented  and  is 
simple  to  use  for  an  estimator.  The  ADPREP  manual  warns  the  estimator 
against  using  the  estimating  equations  outside  of  the  x'ange  of  the  sample 
data.  The  worksheets  insure  that  a detailed  look  is  made  at  every  aspect 
of  the  software  development.  However,  the  ADPREP  technique  also  assumes 
quite  a lot  of  detailed  knowledge  is  available  to  the  estimator.  For 
examplo,  the  estimator  must  know  the  number  of  record  types  in  the  data 
base.  This  is  reasonable  for  the  business  type  of  Army  systems  for  which 


H 


\* 


OSM/SM/76S-4 


the  technique  vu  developed,  However,  for  a real  time  weapon  syetem,  such 
variables  ara  normally  unknown  for  a long  tima  (Dapt  of  Army,  1975). 

Tacolote  Technique,  A fifth  parametric  technique  waa  developed  by 
Tacolota  Research  for  tha  Offioa  of  Naval  Research,  At  first,  tha 
raaaarchara  gathered  a data  baae  of  J87  data  points.  Tha  data  included  the 
169  data  points  from  the  SDC  study  as  woll  as  data  from  TRW,  NAA  Autonotice 

i 

and  12  other  sources*  After  gathering  this  mass  of  data,  tho  researchers 
found  that  it  was  impossible  to  Interpret  the  data  because  much  of  it  waa 
from  older  sources  with  no  available  spokesman  to  interpret  the  meaning  of 
tho  various  data  elements , This  problem  is  common  in  an  area  where  .lack 
of  definition  of  terms  such  as  cost  and  instruction  make  most  of  the 
available  data  essentially  useless  for  research. 

The  Tecolote  researchers  then  abandoned  the  massive  data  baso  and 
sslected  only  flvo  points  about  which  they  had  reasonable  information. 

They  then  proceeded  to  regress  the  number  of  man-months  of  effort  against 
a number  of  different  variables.  They  proposed  their  provisional  technique 
as  an  engineering  scaling  law  rather  than  strictly  derived  statistical 
equations. 

The  Tecolote  effort  produced  a number  of  estimating  equations.  One 
set  of  the  equations  predicted  the  number  of  direct  labor  man-monthe 
necessary  to  totally  develop  a software  package  - from  defining  users 
requirements  to  validating  the  software.  The  sot  of  equations  utilizes 
different  input  variables  to  output  man-months.  For  example,  if  one  knows 
the  number  of  air  threats  a weapon  system  will  be  required  to  track,  the 
following  formula  is  provided: 

1 88 

Total  man-months  labor  = 69  (No,  of  targets)  * 

In  this  case,  the  estimating  parameter  is  a functional  characteristic  of 
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the  weapon  ay o torn  and  not  of  the  software. 

On  the  other  hand,  if  one  knows  (or  can  estimate)  the  number  of 
operating  instructions,  the  following  equation  is  provided: 

1 g t 

Total  man-months  labor  = 2,52  (Thousands  of  instructions)'1'*  H 

This  equation  relates  a software  parameter  to  software  cost. 

The  notable  thing  about  the  Tecolote  effort  was  their  effort  to 
relate  both  weapon  system  parameters  and  software  parameters  to  coot. 

Whilo  the  reliability  of  the  data  la  admittedly  questionable,  the  resulting 
equations  provide  a manager  with  an  indication  of  how  cost  might  vary  with 
operational  weapon  system  parameters.  With  this  type  of  information,  a 
manager  might  seek  to  lower  software  cost  by  trading  off  the  operational 
requirements  of  the  system  (Frederic,  1974)* 

Use  Of  Non-Cost  To  Cost  Technique.  Non-coBt  to  coot  techniques  are 
admirable  in  their  attempt  to  consider  the  many  variables  which  can  influ- 
ence software  cost.  However,  as  can  bo  Been  from  the  five  parametric 
techniques  that  have  been  describod,  research  into  the  various  cost 
estimating  relationships  is  far  from  conclusive.  Whilo  some  techniques 
rely  almost  totally  upon  the  number  of  instructions,  some  researchers  have 
found  little  use  of  the  number  of  instructions  in  predicting  cost.  While 
some  researchers  have  postulated  linear  rolat^onshipaibetween  hoWS^  vari- 
abloa,  others  have  postulated  different  exponential  relationships.  Until 
. more  resoarch  has  been  made  with  controlled  data,  tho  current  parametric 
techniques  provide  their  user  with  little  confidence. 

Perhaps  the  problem  with  the  various  techniques  can  be  better  under- 
stood by  examining  tho  various  programmer  productivity  rates  which  different 
researchers  have  found.  Aron,  for  example,  indicated  that  a programmer  can 
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product  from  1 25  to  500  object  instructions  per  month  based  on  nuserou* 

IBM  projects,  Wolvsrton’s  price  per  instruction  matrix  indicates  a produc- 
tivity range  of  62  instructions  to  312  instructions  with  a center  value  of 
156  Instructions  per  month.  The  SDC  researchers  found  that  the  productivity 
for  object  instructions  ranged  from  10  instructions  per  man-month  to  a high 
of  7142  instructions  per  man-month  on  different  projects.  With  data  which 
Indicates  such  wido  differences  in  productivity  rates,,  it  is  easy  to  see 
why  researchers  have  had  problems  In  reaching  similar  conclusions. 

Again,  whether  the  differences  are  due  to  the  lack  of  controlled  and 
comparable  data  or  whether  the  differences  can  be  attributed  to  numerous 
other  causes  is  still  uncertain.  Until  a major  data  collection  effort  la 
made  of  well  defined  data,  the  wide  apectrua  in  the  findings  of  parametric 
researchers  leaves  many  questions  unanswered. 


A State  Of  The  Art  Assessment 

We  have  examined  five  different  categories  of  software  cost  estimating 
techniques.  Each  of  these  techniques  has  certain  problems  in  its  use  and 
its  accuracy.  These  problems  are  reflected  in  the  software  developments 
of  both  industry  and  government.  Underestimation*  are  oxtrewely  common  in 
moat  developments  while  ovorestimations  are  unheard  of.  While  a number  of 
efforts  have  been  made  to  develop  improved  cost  estimation  techniques,  no 
generally  reliable  softwaro  cost  estimating  technique  oxists. 

This  chapter  has  reviewed  only  a few  of  the  major  software  cost 
estimating  efforts.  For  a more  detailed  Explanation  of  many  research 
efforts  and  techniques,  the  reader  is  referred  to  a master**!  thesis  written 
by  Lois  Morin  under  the  advisorshlp  of  Frederic  Brooks  at  the  University  of 
North  Carolina,  The  thesis  is  well  written  and  quite  comprehensive  it  its 
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asseaaaent  or  various  published  software  cost  estimating  techniques. 
Declalona  And  The  State  Of  The  Art 

The  problems  with  managing  software  are  quite  woll  appreciated*  Soft- 
ware development  has  been  a critical  problem  in  most  weapon  system  develop- 
ments and  is  currontly  a major  topic  of  interest  in  the  systems  acquisition 
business*  Having  seen  the  types  of  docinlonn  which  roly  hoavlly  upon  a 
software  cost  estimate  and  having  aeon  the  state  of  tho  art  of  ooftwuro 
cost  estimating,  porhapa  one  of  the  causes  of  the  software  management  prob- 
lem can  be  better  understood  and  appreciated.  Why  ww  select  tho  wrong 
system,  the  wrong  computer,  or  tho  wrong  contractor  can  ofter  bo  traced 
back  to  a decision  maker’s  inability  to  reliably  estimate  software  cost. 

Why  we  fall  in  the  development  of  major  syutoms  such  as  the  Advanced 
Logistics  System  or  why  wo  are  surprised  by  huge  software  overruns  is  also 
perhaps  partially  oxplalnod  by  tho  inability  to  predict  software  cost. 
Again,  if  wo  seek  to  Improve  the  quality  of  our  decisions  concerning  weapon 
system  softwaro,  then  wo  must  bo  able  to  predict  software  coat, 

Beforo  a manager  can  hope  to  control  the  cost  of  software,  he  first 
must  understand  how  software  coots  are  generated.  He  must  understand  the 
relationship  between  weapon  system  parameters  and  software  cost.  He  must 
bo  aware  of  how  £uid  whether  tho  number  of  instructions  ia  related  to  cost,, 
Until  ho  undei'otands  those  rolationahipo,  hla  control  of  software  cost  will 
be  haphazard  and  unguidod. 
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IV,  Methodology 

Tho  purpose  of  this  research  ia  to  provide  managers,  researchers,  and 
cost  estimators  with  insight  into  the  software  coat  estimating  process. 

To  accomplish  this,  the  software  cost  estimating  environment  at  the  Elec- 
tronic Systems  Division  (ESD)  was  oxamined  to  dotormJ.no  what  techniques 
were  being  used,  how  they  ware  bolng  applied,  and  what  problems  were  being 
encountered.  This  chapter  diacusees  the  methods  used  to  gather  and  analyse 
information  concerning  the  environment  at  ESD, 

Working  Hypotheses 

In  essence,  this  research  is  descriptive  in  nature.  We  need  to  know 
what  the  ourrent  environment  is  in  order  to  live  in  it,  to  understand  it, 
and  to  improve  it.  To  describe  every  feature  of  the  software  cost  esti- 
mating environment  at  ESD  wao  beyond  the  scope  of  this  ten  week  effort. 
Therefore,  working  hypothose*  were  formed  to  focuB  on  the  more  interesting 

I 

and  important  facota  of  that  environment.  Those  working  hypotheses  served 
to  guide  tho  research  and  limit  tho  area  of  investigation. 

Again,  tho  hypotheses  were  only  used  to  guide  and  limit  the  research. 
They  are  not  hypotheses  in  tho  formal,  statistical  sense.  No  attempt  was 
mado  to  statistically  accept  or  reject  an  hypotheses.  Instead,  a guess  was 
made  concerning  the  environment  and  ovldonce  was  gathered  concerning  that 
guess. 

Tho  hypotheses  are  stated  hero  in  "null"  form.  That  is,  the  writer 
expected  to  collect  data  which  would  tend  to  refute  each  of  the  hypotheses. 
Tho  hypotheaos  are: 

a.  There  is  a common  mothod  used  in  estimating  the  coat  of  software. 
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b*  The  persons  responsible  for  making  software  cost  estimates  have 
similar  backgrounds, 

c.  The  software  cost  estimate  is  made  when  most  of  the  software  cost 
drivers  are  known, 

d.  A management  reserve  for  software  is  maintained# 

e.  The  independent  cost  estimate  provided  by  an  outside  agency  3s 

truly  independent,  > > 

f.  Software  estimates  are  challenged, 

g.  The  confidence  intervals  which  software  cost  estimators  place  on 
their  estimates  are  narrow  and  uniform, 

h.  There  is  uniformity  in  the  elements  of  software  cost  addressed  by 
a software  cost  estimate,'  / 

i.  The  contractor's  cost  proposal  indicates  the  method  used  to 
predict  the  cost  of  software, 

j.  Contractors'  estimates  in  response  to  a software  Request  For 
Proposal  ( KFP)  do  not  vary  widely. 

k.  The  type  of  software  cost  data  currently  being  collected  on 
software  contracts  will  support  tho  types  of  cost  estimating 
methods  currently  being  used, 

l.  A primary  management  indicator  of  software  status  is  the  comparison 
of  budgeted  cost  and  actual  cost.  The  percent  complete  of  the 
softwaro  task  is  equal  to  the  ratio  of  actual  coot  to  budgeted 
coat. 

m.  The  software  estimate  does  not  change, 

A llmitod  protest  of  the  Interview  waB  made  at  Wrlght-Pntterson  AFE, 

The  pretest  Identified  some  questions  that  were  vague  and  required  refine* 
ment,  A revised  sot  of  questions  was  prepared  for  use  In  the  EJSD  interviews. 
The  actual  interview  instrument  la  contained  in  Appendix  B, 

The  interview  contains  a large  number  of  questions.  The  reason  for 
this  was  twofold.  First,  the  research  effort  was  attempting  to  describe  an 
environment.  To  adequately  do  that,  as  many  facets  of  that  environment 
which  could  reasonably  be  described  had  to  be  explored.  Second,  the  soft- 
ware acquisitions  at  ESD  are  at  different  stages.  Some  programs  are  not 
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yet  under  contract  and  would  not  be  able  to  answer  question*  concerning 
contractor  performance.  It  was  expected  that  few  of  the  interviews  would 
bo  able  to  address  every  area.  In  order  to  insure  that  an  adequate  amount 
of  information  was  obtained,  a large  number  of  questions  were  developed. 

The  Interview  Subjects 

With  the  interview  instrument  complete,  it  was  then  necessary  to 
identify  the  interview  subjects.  There  are  21  programs  at  ESD  which  involve 
the  acquisition  of  a major  software  package.  With  the  assistance  of 
Major  Richard  Grimm  and  Captain  Gerald  Bourdon  of  the  Cost  Analysis 
Division,  a letter  (Appendix  A)  waa  sent  to  each  program  office  asking 
them  to  establish  a point  of  contact  for  the  interview.  The  letter 
requested  that  the  point  of  contact  be  the  person  most  knowledgeable  of 
the  software  cost  estimate  made  for  each  program. 

Administering  The  Interview 

The  interviews  wore  pei*oonally  admi  nistered  by  the  writer  during  the 
period  12-23  April  1976.  An  attempt  was  made  to  interview  each  of  the  21 
points  of  contact  during  that  period.  However,  the  limited  time  available, 
coupled  with  the  non-availability  of  some  personnel,  resulted  in  only 
12  Interviews. 

In  some  cases  it  was  pos&ible  to  gain  some  limited  information  concern- 
ing a program  without  conducting  an  interview.  This  was  done  primarily  by 
reviewing  contractor  cost  performance  reports  (CPR)  for  those  programs  on 
contract.  Therefore,  with  the  Interviews  and  the  cost  performance  reports, 
some  data  was  gathered  on  16  of  the  21  programs. 

Table  2 indicates  the  programs  at  ESD  which  involve  a major  software 
acquisition  The  nature  of  the  data  collected  is  alBO  indicated.  Note 
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that  the  interviews  were  conducted  in  almost  all  of  the  ESD  depurates  and 
were  not  - limited  to  any  one  elusion  area. 

Administering  each  interview  took  from  50  minutes  to  three  hours.  The 
wide  span  of  time  resulted  from  the  different  nature  of  the  programs.  In 
every  case,  the  interview  subject  was  totally  cooperative. 

Notes  were  taken  at  the  time  of  tho  interview  on  the  responses  to  the 
four  open  ended  questions.  These  notes  were  expanded  at  the  end  of  each 
day. 

Interview  Responses 

Am  expected,  few  of  the  Interview  subjects  were  able  to  address  each 
of  the  question  seta.  In  most  cases,  this  was  due  to  the  nature  of  the 
program.  Again,  subjects  whose  program  was  not  yet  on  contract  could  not 
respond  to  tha  questions  addressing  contractor  performance.  In  other  cases, 
the  interview  subject  was  unfamiliar  with  certain  ampects  of  the  program 
and  unable  to  readily  obtain  the  necessary  information.  The  net  result  is 
that  while  some  of  the  question  sets  were  answered  by  as  many  as  15  sub- 
jects, some  of  the  question  sets  were  answered  by  only  three. 

In  the  chapter  on  research  findings,  some  remeons  are  offered  to 
explain  the  limited  response  for  a particular  question  set.  It  is  hoped 
that  these  reasons  will  indicate  areas  in  which  information  is  sparse  and 
serve  to  guide  future  researchers. 
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>ST  PERFORMANCE 


INTERVIEW 


Airborne  Warning  and  Control  System 

(AWACS)  - YW  YUS  YES 

Advanced  Airborne  Command  Poet 

(AABNCP)  - YS  YES  N/A 

World  Wide  Military  Command  and  Control 

System  (WWMCCS)  - WW  NO  NO 

SAC  Automated  Total  Information 

Network  (SATIN  IV)  - MCV  NO  N/A 

NORAD  Cheyenne  Mountain  Improvements 

(427M)  - MCN  YES  YES 

Air  Force  Enlistment  and  Entrance 

System  (AFEES)  - MCH  YES  YES 

TRACALS  - OCN  NO  NO 

Combat  Grande  - OCW  NO  YES 

Cobra  Dane  - OCD  YES  NO 

Over  The  Horizon  Radar  (ifl^L)  - OCS  NO  YES 

Joint  Surveillance  System  (JSS)  - OCU  PARTIAL  N/A 

Pave  Paws  - OCL  YES  N/A 

Tactical  LORAN  - DCL  YES  YES 

Satellite  Communicatlonn  (AFSATCOM)  - 

DCK  YES  N/A 

TACS  Improvement  (48%)  - DCY  YES  YES 

TIPI  Image  Interpretation  (TIPI  II)  - 

DCM  YES  YES 

TIPI  Tactical  Electronic  Processing 

and  Evaluation  (TIPI  TERPE)  - DCM  YES  N/A 

TIPI  Display  Control/Storago  Retrieval 

(TIPI  DC/SR)  - DCM  NO  YES 

Joint  Tactical  Information  Dissemination 

System  (JTIDS)  - DCB  NO  NC 

Combat  Theater  Communications  - DCJ  NO  NC 

Office  for  the  Application  of  Special 

Intelligence  Systems  (OASIS)  - XRI  YES  N/A 

♦Indicates  that  the  Cost  Performance  Report  was  not  available 
either  because  the  effort  is  not  yet  on  contract  or  that  cost 
reporting  on  software  was  not  a contract  requirement. 

Table  Z 

Sources  of  Data 


h.  *4 


i 


OSM/SM/V6S-4 


Analysis 

With  tho  data  collection  complete,  the  next  ta*.'  wa s to  exaaino  or 
analyze  the  data  to  determine  if  it  tended  to  support  or  refute  the  initial 
hypotheses.  Again,  the  nature  of  this  research  effort  is  descriptive*  No 
causal  relationships  were  hypothesized  or  explored. 

The  technique  used  to  examine  the  data  and  tho  hypotheses  is  quite 
simple.  In  the  following  chapter  on  finding^  each  hypothesis  and  its 
related  question  set  are  examined  individually.  The  hypothesis  is  stated 
and  then  discussed  to  insure  a clear  understanding  of  the  hypothesis  and 
Its  impact. 

After  the  hypothesis  is  understood,  the  question  set  used  to  collect 
data  is  examined,  and  the  data  sought  by  each  of  the  questions  is  explained. 
Since  the  sources  of  data  varied  for  each  question  set,  the  programs  for 
which  data  was  available  are  identified.  Questions  for  which  limited  or 
no  response  was  obtained  are  re-examined  to  explain  why  the  response  was 
limited.  This  may  aid  future  researchers  and  also  serves  to  Indicate 
areas  in  which  information  is  sparse. 

With  the  hypothesis  and  the  question  set  understood,  the  responses  are 
then  examined.  In  some  cases  it  is  only  necessary  to  look  at  some  aggregate 
characteristics  of  the  data  such  as  the  range  and  the  median.  In  other 
cases,  it  is  desirable  to  examine  each  of  the  individual  responses.  The 
technique  used  for  examining  each  set  of  responses  is  adapted  to  the 
peculiar  nature  of  the  questions  and  the  types  of  responses. 

Finally,  with  both  the  hypothesis  and  the  data  understood,  a determina- 
tion is  made  whether  the  data  tends  to  support  or  refute  the  hypothesis. 
Admittedly  this  determination  is  somewhat  subjective.  However,  the  reader 
is  provided  with  sufficient  information  concerning  the  hypothesis  and  the 
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data  so  that  he  can  r«ach  hie  own  conclusion®  concerning  th®  validity  of 
the  hypothesis* 

Again,  each  hypothesis  and  question  set  are  examined  individually* 

Both  failures  and  successes  in  collecting  data  are  explained  to  assist 
future  researchers. 

No  attempt  was  made  to  correlate  the  data  from  one  question  sot  with 
the  responses  from  another  question  set.  Since  most  question  sets  wore 
answered  for  different  subsets  of  programs,  any  such  correlation  would  have 
been  misleading. 
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V,  Research  Fjndln gs 

This  chapter  presents  the  results  of  the  interviews  conducted  at  E3D. 
The  data  collected  against  each  of  the  thirteen  hypotheses  is  examined  and 
analyzed*  Each  of  the  hypotheses  forms  the  basis  of  a research  finding* 

Each  finding  is  addrossod  individually,  using  the  following  format. 

Hypothesis 

For  each  of  the  research  findings,  the  Initial  working  hypothesis  la 
reviewed  and  discussed*  The  impact  of  accepting  or  rejecting  the  hypothesis 
is  examined  in  broad  detail*  Again,  each  hypothesis  is  stated  in  null  form* 
That  is,  the  wrltor  expected  the  data  to  refute  the  hypothesis. 

Sources  Of  Data 

Since  not  all  of  the  subjects  interviewed  were  able  to  address  each 
hypothesis,  the  sources  of  data  for  each  of  tho  findings  is  specified. 

This  should  give  the  reader  a better  feeling  for  the  validity  or  relevance 
of  a particular  research  finding. 

Questions  And  Responses 

For  each  of  the  findings,  tho  interview  questions  which  were  initially 
prepared  are  examined.  Problems  which  were  encountered  with  certain  ques- 
tions are  discussed*  Hopefully,  this  discussion  may  aid  future  researchers 
into  the  area. 

The  responses  for  each  of  the . questions  are  then  presented.  The 
range  of  responses,  the  mean  response,  and  other  appropriate  characteristics 
of  the  data  are  also  presented. 
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Discussion 

After  the  data  for  each  finding  la  presented,  a discussion  examines 
the  Impact  the  data  has  upon  the  Initial  hypothesis.  Again,  no  formal  or 
statistical  test  of  the  hypothesis  1b  performed.  Instead,  the  data  is 
again  examined  to  see  if  it  tends  to  support  or  refute  the  hypothesis. 
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Finding  //X  - Software  Coat  Estimating  Methods 


Hypothesis 

Chapter  III  examined  a wide  variety  of  techniques  which  can  he  uaed 
to  estimate  the  cost  of  software  development.  A research  hypothesis  was 
formed  to  examine  what  techniques  were  being  used  and  how  they  were  being 
applied  at  ESD,  The  hypothesis  was,  "There  is  a common  method  used  in 
estimating  the  cost  of  software," 

Sources  Of  Data 

Data  were  gathered  from  thirteen  programs  representing  a wide  cross 
section  of  ISSD  programs.  Due  to  the  sensitive  nature  of  some  of  the 
responses,  the  individual  programs  are  not  identified. 

Questions 

Two  open-ended  questions  were  used, 

Question  #1.  "Briefly  describe  the  software  acquisition  associated 
with  your  program.  What  is  being  acquired?  When?  How?"  Many  of  the 
programs  at  ESD  have  more  than  one  software  acquisition  underway.  The 
intent  of  this  question  was  only  to  identify  a single  software  acquisition 
to  serve  aB  the  object  of  the  following  questions.  In  cases  where  a pro- 
gram had  more  than  one  software  acquisition,  the  one  selected  for  discussion 
was  the  one  with  which  the  respondent  was  most  familiar. 

Question  $2.  "Describe  how  the  program  office  estimate  of  software 
coot  was  obtained,"  The  thirteen  responses  to  this  question  varied 
greatly  in  both  the  amount  of  information  available  and  the  nature  of  the 
available  data.  Personnel  transfers  frequently  made  contact  with  the 
original  estimator  impossible. 
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Rather  then  simply  classifying  the  responses,  the  wide  variety  of  infor- 
mation collected  merits  examining  each  of  the  thirteen  responses.  Again, 
due  to  the  sensitive  nature  of  some  of  the  responses,  the  programs  are  not 
referred  to  by  name. 

Program  A - The  person  interviewed  had  considerable  previous  work  experience 
as  a programmer  and  as  a supervisor  of  programmers  for  a major  software 
contractor.  Using  this  experience,  ho  was  able  to  analyze  the  total  soft- 
ware package  and  divide  the  software  into  modules.  These  modules  ranged 
in  estimated  size  from  600  instructions  to  35*000  instructions.  The 
average  module,  however,  was  about  3*000  instructions  in  size. 

Having  developed  an  estimate  of  the  number  of  instructions,  the  esti- 
mator then  referred  to  the  SDC  study  which  has  previously  been  discussed 
(Nelson,  1967).  He  did  not  use  the  parametric  equations  recommended  by  the 
SDC  researchers.  Instead,  he  selected  a single  figure  from  one  of  the  SDC 
tables  which  indicated  the  mean  productivity  of  programmers  in  terms  of 
JOVIAL  instructions  per  month.  This  particular  table,  and  tho  associated 
problems  with  the  use  of  its  mean  value,  were  previously  discussed  in 
Chapter  III.  Again,  the  SDC  data  was  based  on  only  15  projects  and  indicated 
a very  wide  variance  in  productivity  experienced  by  different  programs. 
Nevertheless,  the  estimator  selectod  a value  which  indicated  an  average 
productivity  of  3^5  JOVIAL  instructions  per  month.  This  value  appeared 
reasonable  to  him  based  on  his  personal  experience  and  baaed  on  hie  knowl- 
edge of  the  particular  software  package.  Using  this  figure  and  the 
estimated  number  of  instructions,  an  estimated  cost  of  software  was  produced. 
Since  this  program  is  nearing  completion,  the  estimator  was  able  to 
comment  on  the  accuracy  of  his  prediction.  For  example,  his  initial  estimate 
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of  the  number  of  instruction#  in  tho  applications  software  was  160,000, 
Three  years  after  the  estimate  was  made,  the  current  instruction  count  is 
156,000,  The  accuracy  is  Impressive, 

Not  only  was  tho  estimator  accurate  in  his  estimate  of  size,  but  also 
in  his  estimate  of  cost.  The  current  cost  to  complete  is  only  20 & higher 
than  the  original  throe  year  old  estimate.  In  light  of  Inflation,  this 
cost  estimate  is  again  impressive. 

Thus,  despite  the  questionable  source  of  an  average  productivity 
figure,  the  estimator  produced  an  accurate  estimate.  It  should  he  noted, 
however,  that  the  respondent  had  nearly  20  years  of  work  experience  in 
programming.  This  work  experience  might  account  for  hie  accurate  estimate 
of  a multi-million  dollar  software  effort. 


Program  B ~ This  program  called  for  the  development  of  a new  system  to 
replace  an  existing  older  system.  A special  team  was  formsd  for  the  express 
purpose  of  sizing  tho  computer  hardware  necessary  for  tho  new  system.  To 
size  the  hardware,  the  team  first  reviewed  the  softwaro  of  the  existing 
system  and  tho  improvements  expected  of  the  new  system.  By  analogy,  an 
estimate  was  mado  of  the  size  of  tho  software.  This  estimate  was  made  only 
for  the  purpose  of  sizing  the  computer  hardware.  The  team  chief  did  not 
believe  that  tho  estimated  number  of  instructions  was  a valid  parameter  for 
estimating  the  coat  of  tho  software.  Nevertheless,  tho  estimated  number 
of  instructions  was  later  provided  as  the  primary  input  to  another  cost 
analyst  to  dstermine  the  cost  of  softwaro.  Using  parametric  techniques 
such  as  tho  Tecoloto  model,  tho  estimator  predicted  software  cost  almost 
solely  based  on  the  number  of  instructions. 

Again,  tho  team  that  developed  tho  estimate  of  tho  number  of  instruc- 
tions did  so  by  analogy  and  not  by  an  engineering  analysis.  Their  intent 
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waa  only  to  size  the  hardware*  Regardless  of  this,  onco  an  estimate  waa 
available  concerning  the  number  of  instructions,  translating  the  number 
of  instructions  to  a cost  was  almost  inevitable. 

One  possible  reason  why  the  program  jumped  at  the  chance  to  estimate 
cost  based  on  the  number  of  lnutructlona  might  be  found  by  looking  at  the 
previous  softwaro  cost  estimates  for  this  program.  Prior  to  the  estimate 
of  the  number  of  instructions,  an  estimato  of  software  coat  had  somehow 
been  derived.  How  the  number  was  obtained  could  not  be  determined.  However, 
during  a nine  month  period  of  management  review  prior  to  contract  award, 
the  initial  estimate  increased  in  largo  stops  until  the  estimate  was  three 
times  the  original  estimate.  Thus,  when  faced  with  the  opportunity  to 
obtain  an  estimato  which  would  appear  bona  fide,  the  program  office  jumped 
at  the  chance.  Again,  estimating  software  cost  solely  on  the  number  of 
instructions  offers  questionable  accuracy.  However,  it  does  give  the 
estimate  a seemingly  scientific  basis  and  makes  the  estimate  more  readily 
acceptable.  To  challenge  the  now  estimate,  one  would  havo  to  challenge 
the  oatimatod  slzo  of  the  software  - a major  task.  Faced  with  continuing 
challenges  and  changos  in  the  software  coat  estimate,  it  was  to  be 
expected  that  tho  program  office  would  jump  at  the  chance  to  develop  an 
estimate  which,  while  possibly  not  accurate,  would  be  difficult  to 
challenge. 

Program  G - The  software  estimate  for  this  program  waa  prepared  by  a program 
office  engineer  who  was  well  supported  with  engineering  and  design  services 
from  a support  contractor.  A complete  design  of  the  new  system  had  been 
contracted  for  which  identified  seventeen  software  modules  and  their 


function.  By  discussing  each  of  these  modules  with  engineers  from  similar 
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systems,  the  estimator  gained  an  appreciation  of  the  effort  Involved  in 
each  module. 

An  estimate  was  then  prepared  for  each  modulo  in  terms  of  man-months. 

The  estimate  for  each  module  was  given  as  a range.  For  example,  the 
expected  or  most  likely  effort  required  for  a module  might  ha  four  man- 
months.  The  least  likely  or  worst  case  estimate  for  tho  same  modulo 
might  be  six  man-months.  The  estimate  for  the  modulo  was  then,  givon  as 
if  to  6 man-months.  Tho  modules  rangod  in  average  size  from  3 to  12  man- 
months,  Tho  total  oxpocted  value  waH  138  man-monthe  with  a worst  case 
eatimato  of  205  man-months.  An  additional  25%  was  added  to  the  estimate 
to  account  for  undefined  modules  resulting  in  a total  estimate  of  173  to 
256  man-months.  Tills  was  then  converted  into  a cost  rango  of  $830,000  to 
$1,270,000  by  multiplying  by  a "loaded"  man-month.  A loaded  man-month 
includes  not  only  tho  direct  labor  cost  of  programmer's,  but  also  the  over- 
head coats  associated  with  support  personnel  and  facilities, 

1 

When  this  estimate  was  presented  to  tho  independent  estimator,  there 
wus  total  agreement  with  the  mot hod  usod  to  estimate  tho  software  coot. 

The  Cost  Analysis  Division  approvod  the  software  cost  ostimato.  However, 
the  same  approval  was  not  forthcoming  from  higher  management  IovoIb, 

Within  tho  program  offlco,  it  was  folt  that  the  ostimato  did  not  adequately 
toko  into  account  tho  effort  roquirod  to  document  and  toot  tho  software. 

Therefore,  the  program  offlco  increased  tho  original  estimate  to  $2,4M, 
effectively  doubling  tho  worst  case  engineering  ostimato.  Again,  this 
doubling  was  duo  to  the  gonoral  uncertainties  surrounding  software  cost 
and  to  on  uncertainty  concerning  what  cost  dements  wero  addressed  by  the 
original  estimate. 

When  tho  program  office  briefed  their  estimate  at  a higher  lovel,  a 


57 


OSM/SM/76S-4 


recommendation  was  made  and  accepted  to  oven  further  increase  the  software 
•atimate  to  take  into  account  program  uncertainties.  In  effect,  the  moat 
likely  estimate  of  $850,000  was  raised  to  $5*800,000  by  two  management 
levels  who  were  anxious  to  insure  that  sufficient  funds  were  budgeted  for 
the  effort. 

Again,  the  intent  hero  is  not  to  critique  tho  method  used  to  arrive 
at  tho  estimate.  In  actuality,  any  of  the  estimates  may  eventually  prove 
to  be  correct.  The  point  here  is  that  a well  thought  out  and  prepared 
engineering  estimate  was  increaaod  by  the  heuristic  of  doubling  it  not 
once,  but  twice.  While  this  may  Insure  that  sufficient  funds  are  budgeted 


for  this  program,  the  high  estimate  may  cause  other  decision  makers  to 
reach  incorrect  conclusions.  For  example,  the  $5*8H  estimate  may  Indicate 


to  some  planners  that  the  proposed  system  is  not  cost  effective.  The 
large  estimate  may  cause  the  program  office  to  dovelop  a large  software 
staff  from  limited  roaources  when  in  fact  the  effort  may  turn  out  to  be 
only  25%  of  the  estimated  size.  Thus,  while  tho  program's  budget  may  be 


sufficient,  other  software  management  decisions  may  suffer  from  the  inflated 
estimate. 
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Program  D - Thero  was  no  data  available  concerning  the  original  cost 
estimate  for  this  system.  However,  in  the  original  contractor  proposal, 
the  software  was  estimated  to  be  20,000  instructions  in  size.  Despite 
this  small  size,  the  computer  bid  by  the  contractor  had  a core  capable  of 
handling  152,000  instructions,  far  in  oxcobb  of  requirements  indicated  by 
the  estimated  20,000  Instructions.  However,  two  years  after  contract 


award,  tho  20,000  instructions  had  grown  to  174,000  instructions.  This 
increase  in  instructions  by  almost  900%  was  now  overloading  the  core  of 
the  computer  requiring  tho  contractor  to  overlay  the  software. 
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Program  B - Chapter  III  discussed  some  of  the  problem*  that  can  arise  when 
using  the  coat  per  instruction  technique.  Essentially , the  use  of  this 
technique  la  aubject  to  large  error*  due  to  the  poor  definition  of  the 
i,*rms  "coat”  and  "instruction*  aa  well  aa  to  tae  questionable  relationship 
between  coat  and  the  number  of  instructions.  Program  E offered  some  clear 
insight  into  this  problem  a ren. 

To  estimate  the  software  coat  of  this  major  acquisition,  the  program 
office  first  obtained  an  estimate  of  the  £ize  of  the  software  from  MITRE, 

This  estimate  broke  the  software  down  into  functional  module*  ranging  in 
aize  from  200  instructions  to  50,000  instructions.  The  large  size  of  eome 
of  the  modules  reflected  the  limited  time  available  for  developing  the 
estimate  as  well  as  the  uncertainties  associated  with  certain  functions. 

To  insure  that  the  instruction  count  was  not  overly  optimistic  the  program 
office  multiplied  the  MITRE  estimate  by  a factor  of  1.5. 

With  an  instruction  count  in  hand,  the  program  office  then  sought  to 
Identify  an  appropriate  cost  per  instruction  factor  to  use  in  estimating 
coat.  With  the  help  of  a coat  analyst,  they  decided  to  use  the  coBt  per 
instruction  factor  exhibited  by  the  AWAC.S  program. 

In  computing  the  AWACS  cost  per  instruction  factor,  the  lack  of  clear 
and  well  understood  terms  was  evident.  The  AWACS  operational  flight  pro- 
gram had  originally  been  estimated  to  be  158,000  instructions,  but  had 
eventually  grown  to  311,000  instructions  in  size.  While  the  operational 
flight  program  was  an  important  component  of  AWACS  software,  it  was  only 
a small  percentage  of  the  total  AWACS  software  acquisition.  Diagnostics, 
teat  routines,  compilers,  and  other  software  programs  were  also  procured. 
However £ when  developing  a cost  per  instruction  factor,  the  cost  analyst 
only  considered  the  number  of  Instructions  in  the  operational  flight  program. 
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Looking  at  th®  total  software  coat  divided  by  only  the  number  of  instruc- 
tions in  the  operational  flight  program  yielded  the  analyst  two  figures. 

If  one  looked  at  the  originally  estimated  158,000  instructions,  the  esti- 
mated cost  per  Instruction  was  $139*  On  the  other  hand,  if  one  considered 
the  311,000  instructions  which  were  finally  delivered  in  the  operational 
flight  program,  then  the  cost  per  instruction  was  only  $79.  The  analyst 
provided  both  of  these  figures  to  the  program  office. 

The  program  office  was  unaware  that  the  cost  per  instruction  figures 
were  based  on  only  the  instructions  in  the  operational  flight  program. 

They  decided  to  be  conservative  and  use  a figure  of  $150  per  instruction. 

They  then  applied  this  factor  to  all  of  their  instructions  including 
diagnostics,  executive,  communications  processing,  and  operational  soft- 
ware. The  resulting  estimate  was  $33M , a major  software  effort. 

In  a rocent  MITRE  report  (Ashe  et  al,  1975),  the  cost  per  AWACS  Instruc- 
tion is  given  as  a range  of  $6  to  $13.50  per  Instruction.  However,  by 
counting  only  the  instructions  in  the  operational  flight  program,  the  AWACS 
coat  per  instruction  factor  rose  to  a high  of  $139.  This  1000%  to  2000% 
increase  is  rep?  tentative  of  what  can  happen  because  of  inconsistent 
definition  of  terras.  If  the  program  office  had  usod  the  published  $6 
figure  In  the  MITRE  report,  the  $33M  estimate  would  have  been  reduced  to 
S1.32M. 

Again,  the  intent  is  not  to  criticize  the  estimators.  The  lack  of 
common  definitions  in  this  case  led  to  a breakdown  in  communications 
between  the  cost  analyst  and  the  program  office.  Once  this  problem  was 
identified,  rapid  action  was  taken  to  correct  the  error.  The  problem 
he  re  is  that  these  errors  may  not  always  be  discovered. 
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Program  F - This  program  is  currently  concerned  with  engineering  changes 
to  a major  software  system.  The  software  managers  do  not  estimate  soft- 
ware coat  internally.  Instead,  they  first  allow  the  contractor  to  prepare 
a technical  and  cost  proposal  for  each  engineering  change.  They  then 
concentrate  their  effort  on  analyzing  the  contractor’s  software  estimate, 
instead  of  trying  to  compare  it  to  an  internally  generated  estimate. 


Program  G - In  this  case,  the  program  office  cost  estimate  was  prepared 
jointly  by  the  program  office  and  the  ESD  Cost  Analysis  Division.  The 
software  was  divided  into  modules  and  an  estimate  of  the  number  of  instruc- 
tions was  made.  Based  on  a phone  call  to  a former  software  instructor,,  the 
estimators  selected  a productivity  factor  of  ten  Instructions  per  day  per 
programmer.  A software  cost  was  then  computed. 

While  the  source  of  the  productivity  factor  might  seem  strange,  the 
factor  was  obtained  from  someone  experienced  in  the  software  area.  Pew 
published  reports  of  programmer  productivity  for  different  software 
developments  are  available.  Therefore,  estimators  frequently  have  to  rely 
on  their  experience  or  the  experience  of  others  to  determine  a value  for 
certain  factors.  The  resulting  estimate  was  reasonably  accurate.  The 
total  cost  of  software  at  completion  was  only  higher  than  the  initial 
estimate. 

P ix) gram  H - The  estimator  on  this  program  divided  the  software  into  seven 
modules.  He  then  estimated  the  resources  necessary  for  each  module.  The 
modules  were  quite  large  in  size.  For  example,  the  smallest  module 
required  five  man-years.  If  we  assumed  that  a programmer  could  produce 
ten  instructions  per  day,  the  smallest  module  was  12,000  instructions  in 
size.  The  large  modules  may  indicate  a lack  of  detailed  knowledge  of  the 
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software  requirements. 

Program  I - The  estimate  for  this  program  was  prepared  by  the  Rome  Air 
Development  Center.  No  details  on  how  the  estimate  w as  derived  were  avail- 
able to  the  program  office. 

Program  J - When  initially  planned,  the  program  office  did  not  anticipate 
the  use  of  software  in  the  design  of  the  system.  However,  after  contract 
award,  the  contractor  made  a design  decision  which  required  the  use  of 
software.  No  software  estimate  was  prepared  by  the  program  office  and  no 
software  cost  data  was  obtained  by  the  program  office. 

Program  K - The  estimator  for  this  program  used  three  different  techniques 
to  estimate  software  cost.  First,  an  estimate  was  made  of  the  number  of 
instructions  by  MITRE  and  program  office  engineers.  Then  an  engineering 
estimate  of  resources  was  made  for  each  of  the  software  modules.  This 
yielded  the  first  cost  estimate. 

A second  cost  estimate  was  made  by  simply  multiplying  the  estimated 
number  of  Instructions  by  a cost  per  instruction  factor.  The  factor  was 
$44  and  represented  the  previous  experience  of  the  program  office. 

A third  estimate  was  made  by  analogy  with  a similar  project,.  In  all 
three  cases  the  resulting  estimates  were  similar.  The  use  of  three 
different  approaches  provided  the  program  office  with  a higher  level  of 
confidence  in  their  estimate. 

Project  L - This  program  used  two  techniques  to  estimate  cost.  First,  the 
number  of  instructions  was  estimated  by  the  sole  source  contractor.  The 
estimator  then  utilized  the  table  provided  by  the  SDC  study  concerning 
programmer  productivity.  Using  these  two  factors,  an  estimate  of  the  soft- 
ware cost  was  made. 
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A second  estimate  was  made  by  analogy  with  a similar  pilot  effort. 
Engineers  felt  that  the  software  for  the  current  effort  was  4 to  8 times 
larger  than  the  pilot  effort.  Baaed  on  this  analogy,  a second  estimate 
was  prepared.  Both  estimates  were  similar. 

Program  M - No  detail  was  available  concerning  the  program  office’s  original 
estimate.  However,  the  contractor's  cost  proposal  did  provide  some  inter- 
esting information.  The  contractor  divided  the  software  into  modules  and 
into  types  of  effort  per  module  (e.g,  design,  code,  document).  He  also 
estimated  the  number  of  instructions  required  for  each  module.  He  then 
applied  a productivity  factor  of  three  man-hours  per  instruction  for  design, 
code,  and  debug  of  an  instruction.  An  additional  $15  per  instruction  was 
added  to  cover  the  cost  of  software  documentation. 

This  method  is  interesting  because  the  contractor's  productivity 
factor  was  supported  by  data  from  a previous  BSD  contract.  While  the 
productivity  is  much  lower  than  other  published  factors,  it  represents 
measured  experience  on  a similar  program. 

Discussion 

It  was  quite  obvious  that  no  two  of  the  thirteen  programs  used  the 
same  approach  to  estimate  software  cost.  The  estimated  coat  per  instruc- 
tion for  a JOVIAL  Bource  instruction  ranged  from  a low  of  $6  to  a high  of 
$lf;0.  The  detail  in  the  estimated  number  of  instructions  varied  from 
modules  200  instructions  in  size  to  50,000  instructions  in  size.  Pro- 
ductivity figures  ranged  from  three  source  Instructions  per  day  to  fifteen 
source  instructions  per  day.  Not  only  were  different  methods  used,  but 
even  those  programs  which  used  similar  methods  used  quite  different  values 
for  the  various  factors. 
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The  lack  of  a common  technique  maj  cause  a problem  for  a decision 
maker,  Using  two  different  techniques,  similar  software  efforts  can  be 
coated  quite  differently.  The  range  in  cost  for  an  estimate,  depending 
on  the  method  selected,  can  be  as  great  ae  10  to  1,  Therefore,  unless 
two  software  efforts  are  estimated  by  the  same  method,  the  decision  maker 
will  be  uncertain  as  to  the  comparability  of  the  two  estimates. 

The  lack  of  a common  technique  may  be  due  to  a number  of  reasons, 
FirGt,  no  single  technique  has  yet  been  proven  to  be  better  than  any  other 
technique.  Second,  there  is  limited  visibility  into  the  area  of  software 
cost  estimating.  No  common  data  base  exists  where  an  estimator  can  compare 
his  technique  with  those  used  by  other  programs.  Some  techniques  are  not 
well  publicized  and  may  be  unknown  to  Borne  of  the  estimators. 
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Finding  #2  - Software  Estimators 


Hypothesis 

The  educational  and  work  background  of  an  Individual  determines  to 
some  extent  the  types  of  software  cost  estimating  techniques  he  can  readily 
utilize.  For  example,  someone  who  has  never  worked  as  a programmer  would 
find  it  difficult  to  judge  whether  programming  a particular  software  module 
would  be  easy  or  hard.  Also,  someone  with  only  a basic  course  in  Fortran 
would  find  it  difficult  to  architect  or  design  a major  software  system  of 
100,000  Instructions. 

An  hypothesis  was  formed  to  examine  the  backgrounds  of  software  cost 
estimators  at  ESD.  The  hypothesis  was,  "The  persons  responsible  for  soft- 
ware cost  estimates  have  similar  backgrounds."  The  hypothesis  was  formulated 
to  determine  whether  a common  level  of  educational  and  work  experience 
existed  among  ESD  software  cost  estimators.  If  researchers  were  aware  of 
the  knowledge  and  experience  limitations  of  the  ESD  software  cost  estimators, 
they  might  better  be  able  to  develop  a technique  which  could  more  readily 
be  used. 

Sources  Of  Data 

Data  was  collected  from  eleven  individuals  representing  the  following 
program  offices:  48 5b,  TEPPE,  427M,  TIPI  II,  LORAN,  AWACS,  AFEES,  OASIS, 

AABNCP,  Pave  Pawc,  and  Cobra  Dane,  It  should  be  remembered  that  the  data 
was  collected  from  the  person  most  knowledgeable  of  the  software  estimate 
prepared  for  his  program.  Due  to  personnel  transfers,  this  person  may  not 
havo  prepared  the  actual  cost  estimate  for  his  program. 
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, lestiona  And  Responses 


Eight  questions  i«r«  asked  In  this  area.  Eleven  persons  war#  abls 
to  respond  to  all  of  tha  questions. 


Question  #3.  "Briefly  describe  your  educational  and  work  back- 
ground," The  question  revealed  that  all  eleven  individuals  had  college 


degrees  in  the  scientific  or  engineering  disciplines.  Eight  of  the  eleven 
had  masters  degrees.  There  sere  eight  military  personnel  ranging  in  rank 
from  first  lieutenant  to  major.  The  three  civilians  sere  QS-12/13  level. 
Question  "How  sany  years  have  you  been  involved  in  systems 


acquisition?"  figure  1 displays  the  responses  to  this  question. 


Humber  of 
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It  can  be  seen  that  more  than  half  of  the  individuals  had  lees  than  four 
year*  of  systems  acquisition  experience,  while  nine  of  the  eleven  had  less 
than  eight  years. 

Question  #5.  "How  many  years  have  you  been  Involved  in  the  acquisi- 
tion of  computer  software?*1  The  responses  to  this  question  were  essentially 
Identical  to  those  of  the  previous  question.  Apparently  the  individuals 
systems  acquisition  experience  is  identical  to  their  software  acquisition 
experience. 

Question  #6.  "How  many  college  credit  hours  of  software  related 
oourses  have  you  attended?"  Figure  Z displays  the  responses. 


lumber  of 
Individuals 


lumber  of  College  Credit  Hours  In 
Software  Related  Courses 

Formal  Software  Education 
Figure  2 
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The  range  was  quite  large.  Two  individuals  had  no  software  courses  while 
three  had  more  than  thirty  credit  hours. 

Question  #7.  "Have  you  ever  worked  as  a programmer?  How  long?" 

Note  that  this  question  sought  to  identify  the  individuals  whose  full  time 
job  was  progressing.  While  aany  of  the  respondents  could  write  a computer 
program,  only  four  had  ever  worked  full  time  as  a programmer.  The  range  in 
their  programming  experience  was  from  one  to  five  years. 

Question  #8.  "Have  you  ever  directly  supervised  a group  of  program- 
aers?  How  long?"  The  same  four  individuals  who  had  been  programmers  had 
also  worked  as  supervisors  of  programmers.  Their  experience  as  supervisors 
ranged  from  one  to  eight  years. 

Question  #9.  "Have  you  ever  had  any  formal  trsiining  in  cost  estima- 
tion?" None  of  the  eleven  individuals  indicated  they  had  any  such  training. 

Question  #10.  "Have  you  ever  estimated  the  software  cost  of  other 
projects?"  Six  had  estimated  software  cost  on  other  projects.  Five  had 
never  estimated  software  cost  on  another  project. 

Discussion 

Th#  data  indicates  that  there  is  a wide  spectrum  of  experience  among 
the  respondents.  There  were  some  who  had  supervised  programmers,  had 
considerable  software  education,  and  had  estimated  software  coats  on  other 
projects.  On  the  other  hand  there  were  some  individuals  who  had  no  pro- 
gramming experience,  no  software  education,  and  no  previous  experience  in 
estimating  software  cost. 

Only  four  individuals  had  the  type  of  work  experience  one  might  expect 
is  necssBary  to  determine  the  difficulty  of  a software  module  or  to  esti- 
mate the  number  of  instructions  in  a module.  Only  three  individuals  had 
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the  level  of  eoftware  education  one  might  expect  la  necessary  to  design  a 
major  new  software  system.  Only  one  individual  had  both  the  work  experi- 
ence and  education  which  might  enable  him  to  analyze  a major  software 
package  in  sufficient  depth  and  to  reasonably  determine  the  resources 
required  for  software  development. 

In  summary,  it  appears  reasonable  to  reject  the  initial  hypothesis. 
It  appears  that  there  is  a wide  variance  in  backgrounds  with  no  common 
level  of  work  or  educational  experience  among  software  cost  estimators 
at  ESD, 
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Finding  - Software  Coat  Drivers 


Hypothesis 

Thera  are  a large  number  of  factors  which  influence  or  drive  the  coat 
of  aoftware.  Due  to  the  nature  of  weapon  system  procurement,  many  of 
these  factors  may  be  unknown  prior  to  contract  award.  For  example,  prior 
to  contract  award  the  type  of  computer  may  be  an  unknown  factor. 

Despite  these  unknown  factors,  a aoftware  cost  estimator  ia  still 
required  to  estimate  software  cost.  To  do  this  he  must  make  a number  of 
assumptions  concerning  the  unknown  factors.  In  hypothesis  was  formed  to 
examine  what  factors  are  commonly  unknown.  The  hypothesis  was,  MThe  soft- 
ware coat  estimate  is  made  when  most  of  the  software  coat  drivers  are 
known,"  The  ten  factors  or  cost  drivers  examined  were  selected  from  the 
SDC  study  which  indicated  that  these  ten  factors  had  a major  influence 
upon  cost  (Nelson,  1967)* 

Sources  Of  Data 

Two  questions  were  formulated.  The  first  addressed  the  level  of 
knowledge  of  the  ten  factors  prior  to  contract  award.  Data  for  this 
question  came  from  the  following  ten  program  offices:  485L,  427M,  TENPE, 

TIPI  II,  LORAN,  AFEES,  OASIS,  AABNCP,  Pave  Paws,  and  Cobra  Dane, 

A second  question  examined  how  well  known  \-hese  same  ten  factors  were 
known  after  contract  award.  Data  for  this  question  was  limited  to  the 
following  six  programs:  485L,  TIPI  II,  LORAN,  AFEES,  Pave  Paws,  and  Cobra 

Dane, 

Questions  And  Responses 


Two  questions  were  formulated  to  examine  the  level  of  knowledge  of 
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ten  factors  prior  to  and  after  contract  award* 

Mutation  #11.  "At  the  tias  the  j cograra  office  made  its  first  fornal 
(e.g.  Program  Management  Plan)  estinate  of  software  development  costs, 
how  well  known  were  the  following  factors?" 

a.  Type  of  computer  (e.g.  UYK-7>  UNIVAC  1108,  etc.) 

Definitely  known  ^Generally  known  Slightly  known  Unknown 

b.  Configuration  and  types  of  peripherals 

Definitely  known  ^Generally  known  , Slightly  known  ^Unknown 

c.  Memory  and  storage  size 

^Definitely  known  ^Generally  known  ^Slightly  known Unknown 

d.  Operating  systea 

^Definitely  known  Generally  known  Slightly  known  Unknown 

e.  Compiler  and/or  assembler 

Definitely  known  ^Generally  known  Slightly  known  Unknown 

f.  Number  and  typo  of  interfaces 

Definitely  known  ^Generally  known  Slightly  known  Unknown 

g.  Number  of  input  message  types 

Definitely  known  ^Generally  known  Slightly  known Unknown 

h.  Number  of  output  message  types 

Definitely  known  ^Generally  known  Slightly  known  Unknown 

I,  Response  time  requirements 

Definitely  known  ^Generally  known  ^Slightly  known  Unknown 

J.  Number  of  Computer  Program  Configuration  Items  (CPCIs) 

Definitely  known  Generally  known  Slightly  known  Unknown 

To  analyze  the  responses,  different  weights  were  assigned  to  Indicate 
the  extent  to  which  a factor  was  known.  The  following  weights  were 
assigned: 
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Definitely  known  - 10 
Generally  known  - 7 
Slightly  known  - jS 
Unknown  - 0 


j With  these  weights  assigned,  it  was  easy  to  judge  how  well  the  various 

i 


i 


factors  were  known  for  each  program,  Again,  each  of  the  ten  factors  was 
assigned  a weight  between  0 and  10.  Therefore,  a maximum  score  of  100 
would  indicate  that  all  ten  factors  were  definitely  known. 

The  responses  indicated  quite  a range  between  programs.  Some  programs 
were  follow-on  developments  with  most  of  the  factors  well  known.  Other 
programs  were  at  the  other  end  of  the  spectrum.  One  program  had  a total 
score  of  97  while  another  achieved  a score  of  only  10.  The  msan  score  of 
the  ten  programs  was  65.2  with  a median  of  68. 

Thus,  an  estimator  whose  program  had  a score  of  10  had  a great  many 
uncertainties  facing  him  when  a cost  estimate  was  made.  On  the  other 
hand,  the  estimator  whose  program  scored  97  had  only  limited  uncertainty 
facing  him. 

Examination  of  the  individual  factors  yielded  eomo  interesting  results 
The  least  known  factor  was  the  operating  system  which  can  have  a major 
impact  upon  software  cost.  The  best  known  factor  wae  the  compiler.  This 
is  probably  due  to  the  fact  that  a JOVIAL  compiler  is  frequently  specified 
for  most  ESD  programs. 

Question  #12.  This  question  examined  the  same  factors.  However,  the 
lead  part  of  the  question  was,  "At  the  time  of  contract  award,  how  well 
known  were  the  following  factors?”  Again,  data  was  only  obtained  for  six 
programs. 

Using  the  same  weighting  system,  the  programs  which  answered  both 
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question  11  and  12  were  examined.  A marked  improvement  in  the  level  of 
knowledge  waa  obvious.  Prior  to  contract  award  the  six  programs  had  a 
range  of  10  to  97  with  a mean  value  of  62.33.  After  contract  award,  the 
eix  programs  had  a range  from  82  to  100  with  a mean  value  of  86.83.  As 
expected,  once  a winning  contractor  was  selected,  the  factors  which 
influence  software  costr  were  better  defined. 


Discussion 

It  is  apparent  from  the  wide  spread  of  responses  that  the  level  of 
knowledge  of  the  ten  factors  is  dependent  upon  the  nature  of  the  acquisi- 
tion. Those  programs  which  are  acquiring  software  for  an  existing  data 
automation  system  have  few  uncertainties  to  address.  On  the  other  hand, 
those  programs  for  which  little  is  known  about  the  system  face  many 
uncertainties. 

The  mean  score  of  63.2  prior  to  award  indicates  that  for  moat  pro- 
grams many  of  the  factors  are  known  to  some  extent.  As  expected,  the 
award  of  a contract  greatly  increases  the  level  of  knowledge  of  these 
factors. 

In  summary,  the  extent  of  knowledge  of  the  ten  factors  is  dependent 
upon  the  nature  of  the  program.  The  level  of  knowledge  varies  greatly 
between  programs  at  ESD. 
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Finding  $4  - Management  Reserve 


Hypothesis 

In  light  of  the  many  uncertainties  surrounding  a software  cost  esti- 
mate, it  seems  reasonable  to  assume  that  program  managers  would  budget 
additional  funds  for  software  as  a management  reserve.  Tho  following 
hypothesis  was  formulated  to  examine  the  nature  and  size  of  that  manage- 
ment reserve:  "A  management  reserve  for  software  is  maintained. " 

Sources  Of  Data 

Only  eight  of  the  twelve  individuals  interviewed  were  familiar  with 
the  concept  of  a management  reserve.  Programs  for  which  responses  were 
obtained  are:  4 35L,  427M,  TERPE,  TIPI  II,  LORA.N,  AWACS,  Pave  Paws,  and 

Cobra  Dane. 

Questions  and  Responses 

Six  questions  were  administered. 

Question  #13.  "Are  you  familiar  with  the  concept  of  a "management 
resorve?"  Eight  individuals  were  familiar  with  the  concept.  The 
following  five  questions  were  then  administered  to  these  eight  individuals. 
Question  $14.  "Did  the  program  office  establish  a management  reserve 
for  software?"  Seven  of  the  eight  individuals  indicated  that  their  programs 
had  established  such  a reserve.  Sometimes  this  reserve  was  created  by  the 
contractor  within  his  own  budget.  However,  more  frequently  the  reserve  was 
created  by  the  program  office  with  funds  not  placed  on  any  particular 
contract.  Most  of  the  respondents  indicated  that  the  reserve  was  not 
identified  as  such  to  prevent  its  loss  during  a budget  cut. 

Question  $Ib.  "If  yes,  was  the  management  reserve  left  in  the  program 
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until  the  contract  was  essentially  complete?"  This  question  was  poorly 
worded.  It  applied  only  to  programs  which  were  complete  and  it  assumed 
that  the  management  reserve  was  not  used.  No  responses  were  collected. 

Question  #16.  "Do  you  feel  that  a software  management  reserve  should 
be  established  for  each  major  software  acquisition?"  Seven  of  the  respond- 
ents strongly  agreed  with  this  concept.  One  disagreed  feeling  that  such 
reserves  would  tie  up  funds  required  for  other  programs. 

Question  #17.  "Do  you  feel  that  such  a management  reserve  would  be 
approved  during  the  budget  process  at  HQ  AFSC  and  HQ  USAF?"  Four  respond- 
ents felt  that  such  a reserve  would  be  approved  in  light  of  the  frequent 
overruns  in  software  acquisition.  Three  others  felt  that  any  reserve 
identified  as  such  would  not  he  approved. 

Question  $18.  "What  size  of  management,  reserve  (as  a percentage  of 
the  estimated  software  cost)  do  you  feel  a program  similar  to  yours  should 
budget?"  The  responses  ranged  from  5%  to  50%  with  a mean  of  18%. 

Discussion 

It  is  apparent  from  the  data  that  most  programs  do  establish  manage- 
ment reserves.  However,  the  reserve  may  not  be  explicitly  identified  as 
a reserve  due  to  budgetary  pressures. 
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Finding  #5  - Independent  Estimates 


Hypothesis 

A program  manager  sometimes  tries  to  reduce  the  uncertainty  surround- 
ing a software  cost  estimate  by  having  an  outside  agency  prepare  an 
independent  estimate.  At  ESD  these  independent  estimates  are  normally 
performed  by  the  ESD  Cost  Analysis  Division. 

An  hypothesis  was  formed  to  examine  whether  the  Independent  estimate 
prepared  for  software  was  truly  independent.  The  hypothesis  was,  “The 
independent  cost  estimate  provided  by  an  outelde  agency  is  truly  inde- 
pendent. N 

Sources  Of  Data 

Only  five  individuals  were  familiar  with  the  independent  coet  estimate 
prepared  for  their  program.  Data  were  gathered  for  the  following  programs: 
485L,  TERPE,  42 7M,  AFEES,  and  AABNCP . 

Questions  And  Responses 

Six  questions  were  formulated  to  address  this  area.  Unfortunately, 
data  \ as  only  available  for  three  of  the  six  questions. 

Question  #19.  "Was  an  independent  cost  estimate  prepared  for  software?" 
Again,  only  five  individuals  were  aware  of  an  independent  cost  estimate 
prepared  on  their  program. 

Question  #20.  "What  was  the  Independent  estimate?"  This  data  was  not 
readily  available.  Also,  any  figure  which  could  have  been  obtained  would 
have  been  difficult  to  assess.  Often  the  independent  estimate  is  made  years 
before  contract  av/ard.  To  compare  this  independent  estimate  with  the 


current  estimate  would  not  be  meaningful  due  to  the  many  changes  which 
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might  have  occurred  in  the  program*  To  compare  the  independent  estimate 
to  the  program  office  estimate  at  that  time  was  attempted.  However, 
neither  the  program  office  files  nor  the  file*  of  the  Coet  Analysis  Division 
had  this  information  readily  available. 

Question  $21,  "What  era*  the  program  office  estimate  at  that  time?" 
Again,  this  Information  was  not  roadily  available. 

Question  #22. . "Which  eatlmate  do  you  feel  la  more  accurate?"  Again, 
no  responses  were  collected  since  the  relative  estimates  were  not  avail- 
able, 

Question  #23.  "Did  you  provide  the  independent  estimator  with  your 
estimate  of  the  number  of  instructions?"  All  five  program  offices  which 
had  independent  estimates  indicated  that  they  had  provided  the  independent 
estimator  with  their  estimate  of  the  number  of  instructions. 

Question  #2q..  "Briefly  describe  the  type  of  information  you  provided 
to  the  independent  estimator;,"  All  five  of  the  individuals  indicated  that 
the  Independent  estimators  were  given  a complete  description  of  how  the 
program  office  estimate  was  arrived  at.  All  assumptions  and  parameters 
were  detailed  to  tho  independent  estimators. 

Discussion 

Informal  discussions  were  held  with  a number  of  the  analysts  from  the 
ESD  Coat  Analysis  Division.  These  discussions  indicated  that  while  different 
techniques  were  used  to  independently  estimate  software  cost,  all  of  the 
techniques  relied  heaviiv  upon  the  estimated  number  of  instructions  s'lnce 
this  software  characteristic  was  generally  available  from  the  program  office. 
While  the  Independent  estimators  try  to  insure  that  od  adequate  engineering 
estimate  is  made  by  the  program  office  of  the  number  of  instructions,  they 
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do  not  have  the  capability  or  the  resources  to  independently  estimate  the 
number  of  instructions. 

Therefore,  since  both  tha  pro gran  office  astlnata  and  the  independent 
eatimate  rely  upon  tha  sane  aatlmate  of  the  number  of  instruction*,  it 
appears  reasonable  to  reject  the  hypothesis.  Unless  the  independent 
estimator  can  obtain  an  independent  estimate  of  the  number  of  instructions 
for  use  in  his  coot  analysis,  then  the  estimates  he  provides  cannot  be 
considered  truly  independent. 
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Finding  #6  - Challenges  of  Estimates 


Hypothesis 

Since  software  cost  estimating  is  a difficult  and  uncertain  process, 
one  might  expect  that  software  estimates  are  frequently  challenged  as  to 
their  accuracy.  On  the  other  hand,  the  lack  of  a reliable  coat  estimating 
technique  may  prevent  any  serious  challenge  of  the  initial  software  estimate. 
An  hypothesis  was  formed  to  examine  whether  program  office  estimates  or 
contractor  estimates  are  challenged.  The  hypothesis  was,  "Software  esti- 
mates are  challenged." 


Sources  Of  Data 

Data  were  obtained  for  the  following  eight  program  offices:  4851*, 

TERPE,  42 7M,  TIPI  II,  AFEES,  OASIS,  AABNCP,  and  Pave  Paws. 

Questions  And  Responses 

Two  sets  of  questions  were  prepared.  The  first  set  addressed  challenges 
of  the  program  office  estimate.  Eight  responses  were  obtained  for  this  set. 

A second  set  addressed  challenges  of  contractor's  estimates.  Due  to  poor 
question  wording,  no  responses  were  collected  for  this  question  set. 

Quest j. on  #2$.  "Was  the  accuracy  of  the  SPO's  software  cost  estimate 
challenged  by  anyone?"  The  responses  indicated  that  only  one  of  the 
eight  gPO's  had  their  estimate  challenged. 

Question  #26.  "If  yes,  by  whom?"  The  one  challenge  of  a program 
office  estimate  was  made  by  the  Cost  Analysis  Division. 

Question  #27.  "Was  the  accuracy  of  the  contractor's  software  cost 
estimate  challenged?"  This  question  was  poorly  worded  since  the  nature  of 
the  contractual  relationship  between  the  Air  Force  and  the  contractor 
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requires  such  challenges. 

Discussion 

It  is  apparent  from  the  responses  that  most  program  office  estimates 
are  not  challenged.  There  are  a number  of  reasons  for  this.  First,  many 
of  the  program  office  estimates  are  prepared  with  staff  assistance  from 
the  Cost  Analysis  Division,  Second,  for  an  outside  agency  to  question  a 
program  office  estimate,  they  must  have  both  a good  unde  re  tail  ding  of  the 
na+uro  of  the  software  acquisition  and  a good  technique  for  estimating 
software  costs.  Without  these,  one  must  generally  accept  the  program 
office  cost  estimate. 
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When  an  estimator  predicts  a future  cost,  he  does  not  expect  the  actual 
cost  to  fall  exactly  at  that  point  estimate.  Instead,  he  recognises  that  a 
range  of  values  are  possible  and  selects  a value  within  the  range  of  possible 
values*  He  might  select  the  most  likely  or  mean  value  within  the  range* 

On  the  other  hand,  he  may  be  conservative  and  select  a value  at  the  high 
end  of  the  range. 

Since  different  estimators  might  be  using  different  points  within  the 
range  of  possible  values,  an  hypothesis  was  formed  to  examine  what  the 
range  was  and  what  point  in  the  range  was  being  selected.  The  hypothesis 
was,  "The  confidence  intervals  which  software  cost  estimators  place  on 
their  estimates  are  narrow  and  uniform* 

Sources  Of  Data 

Since  the  questions  were  not  applicable  to  those  programs  which  were 
complete  or  nearly  complete,  the  response  was  limited.  Also,  some  individ- 
uals did  not  wish  to  guess  at  how  high  or  how  low  software  costs  might  bs. 
This  may  have  been  due  to  the  wording  of  the  question* 
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answers  is  given  in  terms  relative  to  the  program  office  estimate. 


PROGRAM 

PROGRAM  OFFICE 
ESTIMATE 

CONTRACTOR 

ESTIMATE 

POSSIBLE 

LOW 

POSSIBLE 

HIGH 

PERCENT 

COMPLETE 

Program  A 

A 

0.26A 

1.20A 

1.20A 

60 

Program  B 

B 

N/A 

0.6B 

B 

0 

Program  C 

C 

C 

0.84C 

? 

20 

Program  D 

D 

D 

D 

1.40D 

0 

Program  E 

E 

E 

E 

1.20E 

0 

Table  3 

Range  of  Estimates 
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Question  $30.  "What  was  the  contractor's  estimate  of  software  cost 
in  his  proposal?"  Again,  the  responses  are  shown  in  Table  3.  Note  that 
for  three  programs  (l«e.  C,  D,  and  E) , the  program  office  estimate  and  the 
contractor  estimate  are  the  same.  After  contract  negotiations,  these 
three  programs  felt  that  the  initial  contractor's  estimate  was  a reasonable 
value  and  altered  their  program  office  estimate  to  match  it. 

Note  that  for  Program  A,  the  contractor's  estimate  was  only  26%  of 
ths  program  office  estimate.  While  a buy-in  or  a misunderstanding  is 
indicated,  the  contract  award  was  still  made  at  a prico  which  was  26%  of 
ths  program  office  estimate.  Without  strong  evidence  of  intentional  under- 
bidding, the  pressure  is  upon  the  contracting  officer  to  award  to  the 
lowest  bidder.  The  lack  of  *>  reliable  method  for  verifying  a software 
cost  estimate  may  have  bee.:  -ue  reason  that  sufficient  evidence  could  not 
be  developed. 

Program  B had  not  yet  awarded  a contract. 
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Question  #31.  "How  low  do  you  believe  the  actual  cost  of  software 
might  eventually  be?"  Again,  the  responses  are  indicated  in  Table  3*  Note 
that  two  px’ograas  (l.e.,  D and  E)  consider  the  program  office  estimate  to 
be  the  lower  bound  of  the  range.  On  the  other  hand,  Program  B considers 
the  program  office  estimate  to  be  the  upper  bound  of  the  range. 

Program  A*s  response  indicates  that  tho  program  office  oxpocts  the 
contractor  to  complete  at  120%  of  the  Initial  program  office  estimate. 

This  is  460%  higher  than  the  initial  contract  award.  The  program  office 
is  reasonably  confident  of  this  cost  to  complete  figure  since  a majority 
of  the  contractual  effort  is  complete. 

Question  H- 32.  "How  high  do  you  believe  software  costs  might  eventu- 
ally be?"  Again,  the  data  is  presented  in  Table  3.  The  respondent  for 
Program  C did  not  wish  to  address  this  question/ 

Question  $33«  "What  percentage  of  the  contract  period  is  complete?" 
This  data  is  also  presented  in  Table  3. 

Discussion 

It  is  apparent  from  the  data  that  software  cost  estimators  do  not 
select  the  same  point  within  the  range  of  possible  costs.  Program  B 
selected  a value  at  the  high  end  of  the  range,  Program  0 selected  a middle 
value,  and  Programs  D and  E selected  values  at  the  low  end  of  the  range. 
While  the  limited  data  precludes  any  conclusion  concerning  the  confidence 
interval  surrounding  the  estimate,  the  selection  of  widely  different 
points  within  the  range  of  possible  costs  provides  some  interesting  inf or- 
matlon.  A given  software  cost  estimate  at  ESD  might  represent  the  lowest 
possible  cost,  tho  most  likely  cost,  or  the  highest  possible  cost  depending 
upon  the  particular  program.  Apparently  no  guideline  exists  which  directs 
the  selection  of  a certain  point  in  tho  range. 
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Finding  #8  - Software  Development  Phases 


Hypothesis 

In  preparing  a software  coat  estimate,  on  analyst  must  address  each  of 
the  software  development  phases  or  activities  which  are  applicable  to  his 
program.  While  most  software  acquisitions  include  the  common  development 
phases  or  designing,  coding,  and  debugging  a computer  program,  there  are 
many  other  development  phases.  A software  acquisition  may  include  an 
analysis  of  requirements  prior  to  software  design.  Maintenance  of  software 
is  another  typical  development  phase. 

Most  of  the  software  cost  estimating  techniques  that  have  been  developed 
are  based  on  estimating  the  cost  to  design,  code,  and  debug  software  since 
these  phases  are  common  to  all  software  developments.  An  hypothesis  was 
formed  to  examine  which  phases  were  most  frequently  contained  in  the  soft- 
ware acquisitions  at  ESD.  If  a program  Includes  more  than  the  common  three 
phases  of  design,  code,  and  debug,  then  the  use  of  only  a single  cost  esti- 
mating technique  which  addreasos  only  these  three  phases  would  not  be 
sufficient.  Other  techniques  would  have  to  be  used  to  estimate  the  cost 
of  the  other  development  phases,  AIbo,  if  a cost  researcher  1b  trying  to 
develop  a useful  cost  estimating  technique,  he  should  be  aware  of  the 
various  phases  for  which  a coet  must  be  estimated.  To  oxamine  which  phases 
were  included  in  the  programs  at  ESD,  the  following  hypotheses  was  formed: 
"The  phases  of  software  development  included  in  ESD  programs  nro  common," 

Sources  Of  Data 

Data  were  collected  from  the  following  twelve  program  offices:  48.5L, 

TERPE,  AFSATCOM , 4^7M,  TIPI  II,  LORAN,  AWACS,  AFEES,  OASIS,  AABNCP,  Pave 
Paws,  and  Cobra  Dane. 
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Questions  And  Responses 

Only  one  question  was  used  to  examine  this  hypothesis.  The  question 
■ought  to  identify  which  of  sixteen  software  development  phases  were  appli- 
cable to  a particular  contract. 

Question  #34.  "Which  of  the  following  contractor  tasks  were  included 
in  the  program  office  estimate  of  software  cost?1* 


a* 

Analyze  user  requirement 

Yes 

No 

b. 

Prepare  system  specification 

Yes 

No 

c. 

Define  system  interfaces 

Yes 

No 

d. 

Design  the  data  base 

Yes 

No 

• . 

Develop  program  test  plana 

Yes 

No 

f. 

Specify  all  input  and  output  message 

formats 

Yes 

No 

S. 

Design  and  flow  chart  each  computer  ; 

program 

component 

Yes 

No 

h. 

Write  coded  program  statements 

Yes 

No 

i. 

Compile  and  check  program  code 

Yes 

No 

d ♦ 

Plan  and  run  functional  test  of  each 

program 

Yes 

No 

k. 

Plan  and  conduct  demonstration  test 

Yes 

No 

1. 

Train  user  personnel 

Yes 

No 

m. 

Conduct  demonstration  test 

Yes 

No 

n. 

Assist  in  operational  shakedown 

YaB 

No 

o. 

Develop  software  maintenance  plan 

Yes 

No 

P. 

Maintain  software  after  delivery 

Yes 

No 

The  responses  to  this  question  are  depicted  in  Figure  3.  Some  of  the 
programs  include  all  sixteen  software  development  phases.  One  included 
only  ten  of  the  phases.  On  the  average,  more  than  fourteen  of  the  phases 
were  included  in  a particular  contract. 
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NUMBER  OF 
CONTRACTS  WHICH 
INCLUDE  A 

SOFTWARE  DEVELOPMENT  PHASES  PARTICULAR  PHASE 


Analyze  user  requirement 

6 

Prepare  system  specification 

9 

Define  eyatem  interface 

10 

Design  the  data  base 

11 

Develop  program  teat  plains 

12 

Specify  all  input  and  output 

message  formats 

10 

Design  and  flow  chart  each  computer 
program  component 

12 

Write  coded  program  statements 

12 

Compile  and  check  program  code 

12 

Plan  and  run  functional  test  of 
each  program 

12 

Plan  and  conduct  Integration  test 

12 

Train  user  personnel 

11 

Conduct  demonstration  teat 

12 

Assist  in  operational  shakedown 

12 

Dewelop  software  maintenance  plan 

6 

Maintain  software  after  delivery 

6 

Softwar-o  Development  Phaeos 
. ' ' KU^ce  3 
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Note  that  only  six  programs  included  the  following  three  phaaea: 
analyze  user  requirement,  develop  software  maintenance  plan,  and  maintain 
software  after  delivery.  While  then*  activities  are  performed  by  nearly 
sill  of  the  programs,  they  are  frequently  performed  under  a separate  contrac 
tual  effort. 

Discussion 

It  Is  apparent  from  the  data  the  while  moot  of  the  contracts  havo  many 
common  cost  elements,  some  contracts  include  or  exclude  certain  major  soft- 
ware development  phases.  When  reviewi  ng  a cost  estimate  r *•  BSD,  it  is 
necessary  to  examine  which  of  the  sixteen  phases  are  included  in  that 
estimate  since  there  is  only  limited  commonality  between  contracts.  One 
cannot  assume  that  a given  softwc'  a cost  estlmato  addronsoo  the  same  aoft- 
ware  development  phases. 


87 


*'V"?  ^ L- 

(JSM/SM/76S-4 


Finding  #9  - Cost  Proposal '-, 


Hypothesis 

A contractor's  cost  proposal  can  provide  significant  management  infor- 
mation to  a program  office.  If  a contractor's  cost  proposal  includes  the 
method  used  to  estimate  software  cost,  a program  office  might  be  better 
able  to  determine  if  a contractor  fully  understands  the  nature  of  the 
software  task.  Also,  if  the  contractor’s  estimate  is  baaed  upon  a certain 
programmer  productivity  figure,  this  figure  can  provide  the  program  office 
a method  of  judging  the  status  of  software  development.  For  example,  if 
a contractor  eatlmated  that  his  programmers  would  produce  eight  instruc- 
tions per  day  during  the  term  of  the  contract,  then  any  significant  devia- 
tion from  this  figure  would  indicate  that  the  total  man-hours  required  is 
changing  from  the  initial  estimate. 

An  hypothesis  was  formed  to  determine  the  type  of  software  cost 
estimating  information  which  Is  provided  in  a cost  proposal.  The  hypothesis 
was,  "The  contractor's  cost  proposal  indicates  the  method  used  to  predict 
the  cost  of  software." 


Sources  Of  Data 

Seven  program  offices  were  able  to  provide  data.  The  are:  AFSATCOM , 

427M,  TIPI  II,  LORAN,  AWACS,  AFEES,  and  Cobra  bane,  Othor  programs  were 
either  not  on  contract  or  the  Individual  was  not  familiar  with  the  con- 
tractor's cost  proposal . 

Questions  And  Responses 

Question  $51?.  "Did  the  contractor's  cost  proposal  indicate  how  the 
cost  of  software  was  estimated?"  Four  individuals  indicated  that  the 
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method  for  estimating  software  cost  was  explained  in  the  cost  proposal. 

The  other  three  individuals  indicated  that  there  was  no  supporting  rationale. 

Question  #56.  "If  yes,  briefly  describe  the  method  the  contractor 
used  to  support  his  software  cost  estimate."  Th-ee  contractors  first  esti- 
mated the  number  of  required  instructions.  Then,  they  applied  a programmer 
productivity  factor  to  determine  the  amount  of  diroct  labor  requirod.  In 
one  case,  the  contractor’s  productivity  factor  was  supported  by  data  from 
a previous  contract. 

A fourth  contractor  estimated  cost  by  performing  an  engineering  cost 
analysis.  The  software  was  divided  into  modules  and  each  module  was  esti- 
mated In  terms  of  the  number  of  man-hours  required.  No  estimate  of  the 
number  of  instructions  was  made. 

Discussion 

It  is  apparent  that  the  Inclusion  of  the  software  cost  estimating 
method  in  the  cost  proposal  la  somewhat  random.  When  the  method  is  pro- 
vided, it  does  supply  significant  information.  For  example,  the  estimate 
of  the  number  of  instructions  furnished  by  throe  contractors  could  be 
checked  against  the  program  office  estimate.  Any  significant  difference 
would  indicate  the  need  for  additional  technical  discussions.  Also,  by 
providing  productivity  factors,  the  contractors  provided  the  Air  Force  with 
an  additional  method  for  monitoring  the  status  of  software  development. 

The  three  progi'amB,  which  did  not  receive  any  supporting  data,  x’un  the 
risk  of  awarding  to  a contractor  at  an  unreasonably  low  price.  The  prob- 
lems created  by  such  awards  were  discussed  in  Chapter  II. 
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Finding  #10  - Contractor  BldB 


Hypothesis 

Whan  a contractor  estimates  the  cost  of  software  in  a proposal,  a 
number  of  factors  can  affect  the  bid  price.  Hie  understanding  of  the 
requirements,  his  labor  rates,  and  his  pricing  policy  are  major  factors* 
However,  his  method  of  estimating  software  cost  is  also  an  important  factor. 
While  the  effects  of  theae  factors  are  difficult  to  separate,  it  appears 
reasonable  to  assume  that  understanding,  labor  rates,  and  pricing  policies 
should  not  result  in  radically  different  prices  for  software.  Instead, 
any  radical  difference  is  more  likely  due  to  different  methods  for  esti- 
mating software  cost. 

An  hypothesis  was  formed  to  determine  if  radical  differences  in  the 
bid  price  for  software  do  occur.  While  the  causes  of  these  differences 
are  uncertain,  an  assertion  is  made  that  a primary  factor  is  the  different 
methods  used  to  estimate  software  costs.  The  hypothesis  was,  ‘♦Contractor's 
estimates  in  rssponso  to  a software  Request  For  Proposals  (RFP)  do  not 
vary  widely ,H 
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Sources  Of  Data 

Data  could  only  be  obtalnod  on  three  programs  at  ESD,  Due  to  the 
sensitive  nature  of  contractors*  pricing  data,  ^he  specific  programs 
Involved  ore  not  identified, 

Questions  And  Responses 

Three  questions  were  formulated  which  sought  to  identify  only  the 
highest  and  lowest  bidder  and  determine  the  range  of  the  bids.  However, 
since  data  was  available  for  only  three  programs,  all  of  the  bids  for  a 


i 

i 

i 


{ 

i 

i 

5 

) 
’ i 

4 

j 

i 

4 

t 

i 

j 

1 

1 


OSM/SM/76S-4 


program  are  examined  to  provide  additional  insight. 

Question  $37.  "Was  the  contract  for  software  awarded  competitive! y?" 
All  three  of  tho  contracts  were  awarded  competitively  and  Involved  both 
hardware  and  software.  Only  the  bid  price  for  the  software  line  item  was 

examined* 

Questions  $38  and  $39.  Thooe  questions  only  sought  to  identify  tho 
highest  and  lowest  bidders.  Instead  the  entire  spectrum  of  bids  is 
examined  for  each  of  tho  three  programs. 

The  first  program  had  aevon  bidders,  all  major  defonso  contractors. 

The  Air  Force  estimate  of  software  cost  was  8150,000,  One  contractor  esti- 
mated the  software  cost  ut  one-third  of  this  figure  while  another  estimated 
the  cost  i\t  almost  six  times  higher  than  tho  government  estimate*  The  other 
five  bidders  woro  distributed  evenly  between  these  two  extremes.  Note  that 
the  highest  bid  was  17  times  higher  than  the  lowest  bid.  The  contract  was 
awarded  to  a contractor  whose  software  estimate  happened  to  match  the  Air 
Foret  estimate,  (Note:  This  effort  consisted  of  both  hardware  and  soft- 

ware, Therefore,  tho  bid  price  for  software  was  not  the  only  factor  in 
making  the  award.)  At  the  completion  of  tho  contract,  the  actual  software 
cost  was  fairly  close  to  the  government* a original  estimate. 

A second  program  had  only  three  bidders  with  a much  smaller  range  of 
bido.  If  we  call  tho  government's  initial  estimate  **XM,  then  we  can  examine 
the  bid  prices  in  relation  to  the  government  estimate.  The  three  bids  were 
66%  of  X,  84%  of  X and  126%  of  X respectively.  While  these  bide  bracket 
the  government  estimate,  there  is  still  almost  a two-to-one  latlo  between 
th*  highest  and  tho  lowost  bidders.  Since  this  was  a raulti«millior.  dollar 
ooftware  effort,  tho  difference  was  quite  substantial. 

Data  on  a third  program  was  obtained  from  a briefing  given  by  the  chief 
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of  the  ESI)  Coat  Analysis  Division  v Grimm,  1976).  The  data  provides  the 
responses  of  five  contractor*  who  were  bidding  against  the  same  statement 
of  work.  The  data  includes  the  contractors*  estimates  of  the  number  of 
instructions. 


CONTRACTOR 

INSTRUCTIONS 

COST 

A 

153,000 

$2,800,000 

B 

282,000 

$2,500,000 

C 

^00,000 

$4,600,000 

D 

735,000 

$4,500,000 

E 

766,000 

$2,100,000 

Note  that  the  contractor  with  the  largest  estimate  of  software  size 
(i.e.  Contractor  E)  bid  the  lowest  software  price.  Note  also  the  range 
in  the  sizing  and  in  the  cost.  The  sizes  range  froi*  a low  of  153*000 
instructions  to  a figure  five  times  as  big.  The  coat  range  is  narrower 
but  still  represents  more  than  a fcwo-to~one  ratio  between  the  highest  and 
the  lowest  bidder. 

Discussion 

Again,  the  reasons  for  the  differences  are  uncertain.  It  can  bo  seen 
that  there  is  a wide  range  not  only  in  the  pricing  of  software  but  also  in 
sitting  software.  How  much  of  this  cost  variance  con  bo  attributed  to  tho 
method  used  in  estimating  softwai'e  cost  cannot  bo  determined.  Admittedly, 
some  or  even  most  of  the  difference  might  be  duo  to  misunderstandings  of 
the  requirements.  However,  if  wo  look  at  bidders  D and  E from  the  last 
example,  wo  can  see  that  dospito  reaching  similar  conclusions  concerning 
the  size  of  the  effort,  the  dollar  values  assigned  to  their  efforts  are 
quite  disparate. 
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It  seems  ron«cn*h lo t even  in  light  of  the  limited  data,  to  refute  the 
hypothesis.  Contractor’s  estimates  of  software  coots  do  appear  to  vary 
quite  significantly  when  responding  to  the  same  specifications  and  the  same 

statement  of  work. 
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Finding  /A1  - Availability  o f Data 


Hypothesis 

Finding  #2  described  the  various  types  of  software*  cost  estimating 
techniques  which  are  in  use  at  ESD.  Thee®  techniques  rely  heavily  upon 
such  parameters  as  cost  per  instruction  and  programmer  productivity. 

Since  the  accuracy  of  these  parameters  is  important,  an  hypothesis  ta a 
formed  to  examine  whether  the  contracts  at  ESD  were  collecting  the  type 
of  software  cost  data  necessary  to  verify  these  parameters.  While  verifying 
these  parameters  might  have  only  llmitod  utility  for  the  current  programs, 
it  could  serve  to  gui.de  future  software  cost  estimating  efforts.  The 
hypothesis  was,  "The  type  of  software  cost  data  currently  being  collected 
on  software  contracts  will  support  the  types  of  cost  estimating  methods 
currently  being  used." 

Sources  Of  Data 

Data  was  obtained  for  the  following  eleven  program  offices:  485b, 

42?M,  TERPE,  AFSATCOM,  TIPI  II,  DO RAN,  AWACS,  AFEBS,  AABNCP,  Pave  Paws, 
and  Cobra  Dane, 

Questions  And  Responses 

Eight  questions  were  ashed  to  determine  what  elements  of  information 
were  being  collec.  bed  by  the  eleven  programs.  The  responses  to  each  oloment 
of  information  are  summarised  in  Figure  4. 

Question  $40.  "Does/will  your  contract  call  for  the  reporting  of  cost 
data?"  Ten  of  the  eleven  programs  indicated  thoy  do  receive  contract  cost 


data 


QSK/SM/7bS-4 


CONTRACT  COST 

>A 


SOFTWARE  COST 


MT W 


TOTAL  NUMBER  OF 


INSTRUCTIONS  PER 


MAN-HOURS  FOR 

SOFTWARE 

MAN-HOURS  PER 


MACHINE  HOURS 


MACHINE  HOURS 
PER  CPC 


Type  of 
Information 


Number  of  Programs  for  which  Information 
is  Available  (Eleven  Programs  Interviewed) 


Figure  4 


Question  #i\  1.  MI)oes/will  the  contractor  report  tho  cost  of  software 
as  a separately  identified  item?”  Eight  of  the  eleven  indicated  that  soft- 
ware coats  are  separately  ldeutiflod.  This  In  significant*  Major  programs 
at  the  Aeronautical  Systems  Division  (ASD)  such  as  the  B-l  do  not  separately 
Identify  noftware  in  their  cost  reports.  At  ESD,  however*  the  practice  le 
quite  common. 
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Question  $4?.  "Does/wlll  the  contractor  report  the  total  number  of 
computer  instructions?"  Again,  eight  of  the  eleven  programs  receive  this 

data. 

education  #43.  "Does/will  the  contractor  report  the  total  number  of 
instructions  for  each  Computer  Program  Configuration  Item  (CPCI)?"  (Note: 

A CPCI  ia  a softwaro  modulo  which  includes  a major  augment  of  the  joftwaru 
being  acquired.  For  example,  a compiler  or  an  operational  flight  program 
might  be  CPCI 'a.  If  data  in  available  to  the  CPCI  level,  an  estimator 
might  be  able  to  draw  analogiea  between  aimilar  modules  for  different 
programs).  Seven  of  the  eleven  programs  receive  this  level  of  data. 

Question  #44.  "Doea/will  the  contractor  report  the  total  number  of 
direct  man-hours  charged  to  software?"  This  data  ia  required  to  determine 
programmer  productivity  factors.  Seven  of  the  eleven  programs  collect 

this  data. 

^ueation  #45.  "Does/will  the  contractor  report  the  total  number  of 
direct  man-hours  charged  to  each  CPCI?"  This  information  would  bo  necessary 
to  determine  if  different  productivities  existed  for  different  typos  of 
software  modules.  Only  four  programs  receive  thin  data. 

Queatlon  #46*  "Does/will  the  contractor  report  the  total  number  of 
machine  hours  for  software  development  and  test?"  Computer  time  can  bo  a 
significant  portion  of  softwaro  cost.  While  some  of  the  ESP  programs 
provide  this  resource  as  a government  furnished  itom,  othern  are  charged 
for  machine  hours  by  the  contracto.  , 'To  be  able  to  compare  cost  data  from 
two  programs,  an  analys ; would  have  to  know  whethor  or  not  this  cout  is 
included  in  the  total  software  cost.  However,  only  five  of  the  eleven 
programs  collect  this  data. 

question  # 47.  "Does/will  the  contractor  report  the  total  number  of 
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machine  hours  required  for  the  development  and  tost  of  each  CPCI?'*  Only 
two  programs  received  this  level  of  detail, 

PiscuBBion 

The  collection  of  data  by  programs  at  ESD  is  far  from  uniform.  Some 
programs  collect  all  of  the  above  data  elements  while  others  collect  only 
a few  or  oven  nona  of  them.  To  develop  a reliable  software  cost  estimating 
data  base,  controlled  and  well  defined  data  should  be  collected  from  all 
programs. 

Many  of  the  Individuals  reported  that  while  the  data  was  not  formally 
collected,  it  might  be  obtainable  from  the  contractor.  However,  unions  a 
specific  and  well  defined  aet  of  software  cost  elements  are  contractually 
required  and  formally  reported  for  all  programs,  the  development-  of  a 
comprehensive  and  producti\e  software  cost  data  base  is  extremely  difficult. 

In  general,  it  appears  that  many  programs  at  ESD  are  collecting  the 
types  of  data  necessary  to  develop  a software  cost  estimating  data  base. 
Unfortunately,  some  serious  gaps  exist  In  the  data  collected.  There  is  no 
common  format  or  definition  of  variables  for  collecting  this  data.  Also, 
there  is  no  common  repository  for  the  data  that  is  being  collected.  Unless 
this  data  is  uniformly  collected  and  analyzed  for  many  programs,  the  utility 
of  the  currently  collected  data  in  developing  bettor  cost  estimating  tech- 
niques is  minimal. 
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Finding  #12.  - Software  Status 


Hr,  rotheslc 

Because  of  the  lack  of  well  defined  technical  milestones  for  aoftwaro 
development,  managers  havo  difficulty  in  assessing  the  statue  of  software. 
Nevertheless,  tho  status  muat  ho  determined  for  management  purposes  such 
a m determining  the  amount  of  progrosB  payments  to  allow  a contractor.  She 
status  of  software  is  frequently  given  by  e.  single  parameter  - the  percent 
complete.  While  the  validity  of  Buch  a parameter  may  be  questioned,  it  ie 
in  widespread  use  at  BSD. 

An  hypothesis  was  formed  to  try  to  determine  how  the  percent  complete 
of  software  was  computed.  The  hypothesis  was,  '“A  primary  management  indi- 
cator of  software  status  is  the  comparison  of  budgeted  cost  and  actual  cost. 
The  percent  complete  of  the  software  task  in  equal  to  tho  ratio  of  actual 
cost  to  budgeted  cost."  In  Chapter  II  the  problem  of  using  this  ratio  as 
a status  measurement  was  discussed.  If  the  software  cost  estimate  is  in 
error,  thon  a manager  may  well  be  monitoring  financial  activity  instead  of 
technical  progress. 

Sources  Of  Data 

While  it  was  possible  to  obtain  data  concerning  the  actual  and  budgeted 
cost  of  software  for  many  programs,  it  was  not  possible  to  rondily  collect 
data  concerning  what  percent  complete  the  software  was  jud&od  at  any  upecific 
time.  While  such  data  may  exist  in  the  contracting  officer**  files,  the 
individuals  interviewed  were  not  able  to  provide  thin  information.  There- 
fore, no  data  was  collected  concerning  this  hypothesis. 
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Finding  $13  - Changes  In  Softwaro  Coat  Estimates 


Hypothesis 

After  making  an  initial  estimate  of  software  coat,  a manager  will 
frequently  obtain  additional  information  which  requires  a revision  of  the 
cost  estimate.  An  hypothesis  was  formed  to  examine  when  and  how  these 
changes  occur  during  a program's  lifetime.  The  hypothesis  was  that  the 
software  estimate  did  not  change. 

Sources  Of  Data 

Data  was  gathered  from  ten  programs  at  ESD*  Five  of  these  programs 
were  complete  while  the  other  five  are  still  ongoing.  Due  to  the  sensitive 
nature  of  the  data,  the  individual  programs  are  not  identified. 

Questions  And  Responses 

Initially,  a set  of  eleven  questions  was  formed  to  address  this  hypoth- 
esis, The  questions  appear  in  Appendix  B as  Questions  51  through  61, 
Sufficient  response  was  not  obtained  on  these  questions  for  two  reasons. 
First,  the  questions  assumed  that  each  program  office  maintained  an 
historical  record  of  tho  initial  software  cost  estimato  and  of  the  changes 
to  the  estimate  during  the  lifetime  of  the  program.  Such  data  was  not 
consistently  recorded  by  the  program  offices.  Second,  the  questions  were 
poorly  worded.  For  example,  one  question  asked  for  uhe  cost  estimate  at 
the  time  of  the  Critical  Design  Review  (CDR1,  It  was  thought  that  the  CDR 
was  a single  point  in  time  common  to  most  programs.  Unfortunately,  while 
the  CDR  1b  common  to  most  programs,  it  sometimes  covers  a considerable 
period  of  time  and  is  not  a single  point  in  time.  Therefore,  the  software 
estimate  during  the  period  of  CDR  sometimes  changes  quite  significantly. 
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In  light  of  the  unavailability  of  roeponaee  to  the  eleven  questions, 
an  alternative  source  of  data  war  used.  Many  of  the  contracts  at  BSD 
receive  periodic  reports  from  contractors  concerning  the  progress  of  a 
contract.  One  of  these  reports  is  the  Cost  Performance  Report  (CPR).  The 
CPR  reports  the  contractor’s  progress  against  an  initial  financial  plan. 

A budget  or  financial  plan  is  established  for  each  mayor  task  at  the  start 
of  the  contract.  As  the  contract  progresses,  the  contractor  reports  hie 
actual  expenditures  and  explains  any  variances  between  bis  estimated 
expenditure  and  his  actual  expenditure. 

While  the  CPR  contains  many  items  of  Information,  only  two  were 
relevant  to  this  hypothesis.  The  first  item  is  the  contractor’s  actual 
coot  expended  for  software.  This  actual  cost  includes  direct  labor  and 
overhead.  The  second  item  is  the  contractor’s  estimate  of  total  software 
cost.  As  the  contract  progresses,  the  contractor  reports  both  his  increas- 
ing actual  cost  and  any  changes  which  he  may  make  to  his  estimate  of  the 
total  software  cost. 

Completed  Programs.  We  can  first  look  at  tM  data  which  was  obtained 
from  the  five  programs  whoso  contracts  are  completed.  Figure  5 depicts 
the  type  of  information  gathered  from  Program  A. 
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Th#  horizontal  axis  of  the  graph  deplete  the  numbor  of  month*  elapsed  aftor 
contract  award.  The  vertical  rods  depicts  the  ratio  of  estimated  or  actual 
cost  to  final  cost.  Two  plots  are  made.  One  is  the  ratio  of  actual  cost 
to  final  cost  over  time.  The  second  is  the  ratio  of  the  estimated  cost  to 
final  cost  over  time.  For  example,  Figure  5 indicates  that  th*  initial 
software  cost  estimate  was  only  5?-%  of  the  final  soivjuro  coat*  Also,  tho 
graph  indicates  that  tho  actual  cost  equaled  the  initial  eatimato  when 
only  22  of  the  44  months  of  contractual  effort  wore  complete. 

Some  interesting  observations  can  be  made  from  Figure  5 which  is 
representative  of  the  data  obtained  for  all  flvo  completed  programs. 

First,  one  notes  that  the  ratio  of  actual  cost  to  final  cost  is  essentially 
linear.  The  implication  is  that  tho  level  of  resources  applied  to  software 
is  level  throughout  the  period.  Note  that  for  Program  A,  the  contractor 
probably  sized  the  initial,  software  team  based  on  his  initial  estimate  of 
software  cost.  However,  this  estimate  was  only  52%  of  the  final  coat. 

When  th*  contractor  began  to  realize  that  his  initial  estimate  was  faulty, 
he  did  not  take  any  action  to  significantly  increase  tho  amount  of  resources 
applied  to  software. 

Two  reasons  can  be  postulated  for  the  contractor's  failuro  to  increaso 
software  resources.  First,  increasing  resources  adds  significant  coot  and 
problems  to  the  project.  The  existing  software  team  must  normally  stop 
software  production  while  new  stuff  members  are  trained  and  oriented  to 
th*  new  project.  Second,  the  contractor  does  not  quickly  realize  that  his 
initial  estimate  waB  grossly  wrong.  For  example,  after  16  months  the 
Program  A contractor  finally  raised  his  estimate  of  software  cost.  However, 
the  increased  value  was  only  72%  of  final  cost  and  was  not  significantly 
higher  than  the  original  estimate.  Later,  in  the  28th  month,  tho  contractor 
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again  raised  his  estimate  of  software  coat.  He  now  knew  that  hie  initial 
estimate  was  grossly  wrong,  but  it  was  too  late  to  make  any  significant 
improvement  in  schedule  performance  by  adding  more  resources. 

In  essence,  it  appears  that  the  contractor  has  the  attitude  that  good 
times  are  Just  around  the  comer.  While  his  estimate  might  be  grossly 
wrong,  he  learns  of  this  gross  error  only  slowly.  The  increases  In  tho 
estimate  never  appear  to  be  large  enough  to  merit  the  application  of 
additional  software  resources.  The  net  effect  1b  that  Program  A softwaro, 
which  was  expected  to  take  22  months  and  cost  a certain  amount,  wound  up 
taking  kk  months  and  coating  almost  twlco  os  much. 

The  Impact  of  this  attitude  is  twofold.  There  is  of  course  the  addi- 
tional cost  of  software*  However,  this  additional  cost  might  be  minor  in 
impact  compared  to  the  Indirect  cost  of  a 22  month  schedule  slip.  Such  a 
slip  can  gonerato  indirect  costs  which  well  exceed  the  additional  cost  of 
software. 

A second  observation  can  be  made  from  the  Figure  3 data.  Note  that 
the  estimated  cost  of  software  appears  to  change  in  Irregular  steps.  Again, 
two  reasons  can  bo  postulated  for  this.  First  the  contractor  has  no  method 
for  easily  updating  hie  initial  cost  estimate,  Ilo  does  not  generally  have 
a cost  model  which  can  be  quickly  updated  to  take  into  account  current 
performance.  Instead,  reestimating  is  a major  tank.  The  software  pro- 
duction Is  halted  while  a new  software  dovelopmont  plan  is  formulated  and 
resources  are  estimated  against  this  now  plan.  This  type  of  major  re- 
estimating  effort  can  compound  the  problem  of  rising  software  cost.  There- 
fore, contractors  tend  to  avoid  reestimating  until  it  is  blatantly  obvious 
that  the  initial  plan  Is  faulty. 

The  eoconu  reason  for  the  step-like  bohavior  of  tho  estimated  cost  data 
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1*  that  the  contractor  frequently  1*  uncertain  that  his  current  estimate 
1*  In  error  until  hla  actual  coat  approaches  Ills  estimated  coat  and  the 
software  is  atill  incomplete.  Note  that  for  Program  A the  actual  cost 
almost  equaled  the  estimated  cost  in  the  2?th  month  finally  forcing  the 
contractor  to  realize  the  need  for  r&estlmation, 

A third  observation  can  he  mado  by  looking  at  the  chan go a in  the  uoft- 
ware  estimate  during  tho  surly  months  of  the  oontruct.  One  wight  oxpoct 
that  after  the  software  team  has  been  assembled  and  working  for  throe  to 
six  months  that  they  would  then  have  a valid  ooncoptlon  of  the  size  of  tho 
software.  However,  looking  at  Figure  !?,  we  can  note  that  the  estimate 
actually  dips  In  the  first  months  - increasing  tho  estimating  error.  Not 
until  16  months  into  the  program  Is  a significant  increase  in  the  software 
cost  rocognized.  Apparently,  the  first  months  of  a coxxtract  are  a period 
of  optimism  during  which  any  Indications  of  higher  software  costs  are 
elthor  not  available  or  are  ignored. 

Appendix  C contains  the  graphs  of  four  other  completed  programs 
(Programs  13,  C,  1),  and  Ji),  While  the  initial  coet  estimate  for  these 
programs  varied  from  30%  to  78%  of  tf  i«l  cost,  tho  general  pattern  of  all 
five  programs  is  somewhat  similar,  Ooftwaro  oouts  are  accumulated  linearly; 
software  ostimateu  change  in  irregular  steps;  and  no  major  change  occurs  in 
the  estimate  early  in  the  program. 

Uncompleted  Programs,  Having  looked  at  how  complohod  programs  per- 
formed, we  now  turn  to  tho  five  uncoaplotod  programs.  Figure  6 displays 
the  data  gathered  for  Program  F, 

Note  that  since  tho  programs  aro  still  ongoing,  no  final  cost  of  soft- 
ware is  known.  Therefore,  the  vertical  axis  of  Figuro  6 represents  the 
dollar  value  of  the  ootimated  or  actual  i ost  of  software*  Noto,  ,,'or  oxample, 
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that  the  initial  cost  for  software  for  Program  F was  about  50M.  After 
45  months  of  sffort  the  estimated  coat  to  complete  has  grown  to  $1?M,  more 
than  double  the  original  estimate. 

Again,  the  same  three  observations  can  be  made  for  the  uncompleted 
programs  that  were  made  for  the  completed  programs.  For  example,  the  actual 
comt  are  again  being  accumulated  in  a linear  fashion.  Also,  the  changes  in 
software  estimates  appear  to  occur  in  irrogular  stops.  And  again,  no  major 
change  in  tho  software  estimate  is  ovident  for  tho  first  16  months  of  the 
contract. 

Appendix  D contains  the  data  collected  for  both  tho  completed  and 
uncompleted  programs  graphed  in  the  same  fashion  as  Figure  6,  Tho  graphs 
indicate  different  values  and  different  sizes  of  software  orrora,  Uowsvor, 
all  ton  of  the  programs  tend  to  oxhibit  tho  same  three  propertied  that  wore 
observed  for  Program  A and  Program  F, 

Discussion 

Again,  the  Cost  Performance  Hoport  data  appears  to  indicate  a numbor 
of  things  about  uoftwars  coot  entimaton,  Tho  implications  that  those 
observations  can  havo  for  managomonfc  aro  important. 

Linear  Cost  Accrual,  The  data  indicate  that  none  of  tho  ten  contractorii 
ever  significantly  altorod  the  size  of  the  original  software  team,  Tho 
contractor  will  normally  keep  the  initially  forced  team  working  until  the 
software  is  eventually  completed. 

This  implion  that  any  orror  in  ontimating  software  cost  will  eventu- 
ally bo  translated  into  a ooftwaro  achodulo  slip.  For  oxample,  in  Program  A 
the  contractor  ox'iginully  estimated  software  to  cost  51, 7M  and  take  22  months. 
At  tho  end  of  tho  contract,  .software  cost  had  risen,  to  53,4M,  Since  the 
program  wao  a large  hardware  and  software)  development,  the  dollar  incx-euse 
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in  software  had  only  a omall  impact*  However,  tho  software  schedule  waa 
extended  from  2.2.  months  to  4A  months.  The  indirect  cost  of  this  schedule 
slip  cannot  ho  determined*  However,  it  waa  quito  evident  from  discussions 
with  program  office  personnel  that  the  schodule  slip  had  a much  greater 
Impact  than  the  Increased  cost  of  software. 

Thus,  if  managers  view  tho  completion  ochodulo  of  thoir  program  a u a 
critical  factor,  thon  managors  should  bo  oqually  concerned  with  tho  ootl« 
mation  of  software  cost.  Any  orror  in  tho  software  cost  estimate  will 
probably  bo  translated  into  schodule  slippage, 

Stop  Chan goo  In  Kstlmatos.  One  might  oxyect  that  if  a software  manager 
■aw  that  a certain  software  modulo  had  boon  grossly  underestimated,  he  might 
take  action  to  revise  his  ontimato  of  the  remaining  modules.  In  fact,  the 
software  manager  does  not  appear  to  quickly  react  to  any  ouch  indications. 
Only  whon  tho  actual  coat  of  softwaro  approaches  tho  ostimatod  cost  does 
tho  software  manager  appear  to  bo  sufficiently  motivated  to  stop  producing 
software  and  to  revise  his  software  estimate.  This  typo  of  behavior  in 
demonstrated  in  tho  irregular  ntopwiso  changes  in  the  software  estimate. 
Marly  Optimism,  It  also  appears  that  software  managers  ax‘e  quite 
optimistic  during  tho  early  months  of  a contract.  Again,  one  mxght  expoct 
that  daring  the  early  months  a mox^e  do  finite  software  development  plan  is 
established  which  accuxvxtoly  x'oflects  tho  total  effort  required.  Given 
much  more  time  and  roaoui’coa  than  wore  available  during  tho  preparation 
of  contract  proposals,  one  might  oxpoct  that  the  new  plan  during  the  early 
months  would  lndicato  tho  need  to  accurately  X'ovlae  resources. 

In  fact,  such  a revision  of  software  estimates  doon  not  appear  to 
occur  during  the  oax’ly  months  of  the  contract.  In  many  cases,  the  estimate 
of  softwaro  cost  actually  drops  and  becomes  woroo.  Not  until  quite  late 
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in  the  contract  does  tho  contractor  slowly  and  gradually  reolizo  the  error 
in  his  initial  estimate* 

The  Qood  And  The  Bad.  None  of  tho  five  completed  programs  ever  had 
the  final  cost  of  software  lower  than  the  initial  estimate.  The  heat 
performance  was  from  two  programs  which  exhibited  cost  growths  of  about 
50%,  If  ono  conoidero  that  these  estimates  wore  made  for  a two  yoar  period 
during  which  inflation  was  qulto  high,  then  the  accuracy  of  theue  two 
program  estimates  in  remarkable.  On  the  other  hand,  thoro  aro  other  pro- 
grams whose  software  cost  has  doubled  and  continues  to  climb.  In  short, 
ths  range  in  program  performance,  as  indicated  by  the  initial  software  cost 
estimate,  is  quite  broad.  In  light  of  the  vastly  different  software  oust 
estimating  parameters  and  techniques  in  use,  as  px'osentvd  in  F'indAng  #1, 
the  wide  disparity  of  program  performance  is  not  .'lurpriaing* 
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| . In  1976,  J.'dD  managers  will  bo  making  decisions  concerning  the  acqui- 

j 

J sition  0 f an  eatimated  13  billion  of  software*  Since  coot  je  frequently 

the  dominant  criteria  in  theae  decialono,  the  capability  to  provido  thooo 
managers  with  reliable  and  aoourato  ooftwaro  coot  ootirautco  in  oaoontiul, 

| 

j This  research  effort  haa  provided  some  limited  insight  into  the  (soft- 

ware cost  estimating  process  at  BSD,  Daned  upon  tho  rooearoh  findings, 

i 

there  appear  to  be  aome  major  problem  areas  which  inhibit  the  development 
of  aoourato  and  reliable  software  ooet  estimates,  This  chapter  reviews 
theae  problem  areaa  and  recommends  actions  to  Improve  the  aoftware  ooat 

1 estimating  process, 

, While  this  research  was  limited  to  tho  KBD  environment,  the  aame  types 

; of  problems  may  oxiat  at  other  Do I)  software  aoquialtlon  activities.  There- 

fore, the  adoption  of  theae  recommendations  by  other  Dol)  agencies  should 
also  be  considered. 
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Program  Office  MKst^.mats«ni-in^n_uAaB,eB*ffiOwt; 

One  part  of  the  software  coot  eatimating  process  which  can  he  improved 
ia  the  development  of  internal  program  office  estimated  of  software  coot. 
Before  examining  poouible  improvements , it  ia  important  to  understand  the 
nature  of  the  problems  eatimatoro  at  Kill)  have  In  developing  acorn  *-'e  and 
reliable  estimated, 

Brror  Moure  on  „ in  eotimatlng  aoftware  ooat,  there  are  three  pOB«i.‘)» 
aourcea  of  error.  The  first  souroo  of  error  1*  duo  to  tho  element  of  chano» 
In  the  future  which  makes  ooat  a random  variable.  Ho  matter  how  good  an 
eatimating  technique  might  be,  the  random  nature  of  futuro  coot  will  always 
be  a source  of  error.  Ho  action  can  bo  taken  to  completely  eliminate  this 
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sourco  of  orror, 

A second  sourco  of  error  is  the  estimating  tochniquo  itself.  No 
current  tochnlquo  has  boon  demonstrated  to  consistently  produce  roliable 
and  uccurato  estimatos.  Current  techniques  fail  to  precisely  and  completely 
describe  tho  relationship  that  oxisto  between  co.jt  and  other  paromotoro. 
Until  future  research  develops  better  techniques,  this  source  of  error 
will  continue  to  troublo  program  oifico  estimates. 

Thors  is  a third  sourco  of  orror  which  nuy,  in  tho  writers  opirJon, 
contribute  more  error  than  tho  other  two  sources.  This  third  source  is 
tho  non«unlforra  and  unskilled  application  of  a cost  estimating  technique. 
Regardless  of  how  good  a technique  is,  an  accurate  estimate  may  only  bo 
obtained  if  tho  technique  is  proporly  usod,  Tho  research  findings  Indi- 
cated that  a number  of  problem  ax'oas  exist  which  give  rise  to  this  type 
of  error. 

Problem  Areas,  For  example,  instead  of  using  a common,  well-defined 
method  for  estimating  software  cost,  Finding  //l  indicated  that  MUJJ  esti- 
mators use  u myriad  of  different  and  poorly  defined  techniques.  Cost 
parameters  vary  w.'dely,  Managers  find  it  difficult  to  compare  estimates 
prepared  by  different  techniques.  They  also  find  :l  t difficult  to  Judge 
tho  relative  merits  of  the  various  techniques  since  not  enough  programs 
uso  the  sumo  methods  and  tho  same  cost  parameters 

The  many  different  techniques,  coupled  with  some  unskilled  estimators, 
often  can  result  in  errors.  In  one  caso,  an  estimator  input  an  eetlmato  of 
<?.Q,(X>0  instructions  into  a cost  modo.l  only  to  find  lator  that  tho  tx'uo 
number  of  instructions  was  17k, OOO,  Anothov  ootimauor  intorviowod  did  not 
understand  the  difference  betwoon  source  and  object  instructions.  In  a 
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third  case,  a mlBundorotanding  of  terms  resulted  in  a $33  million  estimate 
for  a software  project  probably  a tenth  of  that  aizo.  These  types  of  mis- 
taken can  result  tr  moro  not  error  than  any  other  source. 

Other  problems  exist.  Finding  #7  indicated  that  a given  estimate 
might  roprooont  ol.thor  the  lowest  possiblo  cost  or  the  highest  possible 
cost.  Finding  //U  indicated  that  all  oetimutos  do  not  addrosa  the  same 
phases  of  coftwaro  dovolopmont. 

Facod  with  thooo  problomu,  managox’s  oeok  outside  help  to  bolster  their 
confidence  in  an  ontimato,  Howovor,  Finding  #5  pointed  out  that  tho  curront 
Indopondent  ootimutoo  wore  strongly  dopondont  on  program  office  estimates 
of  the  number  of  instructions.  The  independent  oatimato  is  likely  to 
suffer  fx'om  the  numo  sources  of  orror  as  tho  program  office  estimate. 

Finding  tt()  determined  that  moat  software  cost  estimates  aro  unchallenged. 
Managers  cannot  count  on  errors  being  dotootod  by  some  outside  agency, 

Ono  mujox  cause  of  those  px'Obloms  was  possibly  idontifiod  by  Finding 
//<'.,  Tho  software  cost  ostlmators  at  F. .'!)_)  do  not  possess  a common,  minimal 
love!  of  work  or  educational  oxporionoo  which  might  qualify  them  to  properly 
analyze  a major  softwaro  effort  and  ontimato  software  coot.  While  a few 
wore  wall  qualified,  others  had  almost  no  wox’k  or  educational  background 
in  software  development. 

An  Assessment,  based  on  tho  research  findings,  it  appeare  reasonable 
to  conclude  that  the  curront  px'oblemo  in  the  software  cost  estimating  process 
at  BSD  are  Inhibiting  the  development  of  the  accurate  and  rol:l.ab?,e  eetimatos 
which  managers  need  to  mako  good  software  managomont  decisions, 

Program  Office  Kntimatos  - A Hocommondatlon 

Ono  approach  to  minimizing  the  above  problems  is  to  havo  all  USD 
program  officos  utilize  a common,  sound,  softwaro  cost  estimating  tochniquo. 
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Based  upon  this  research,  it  Is  recommended  that  ESD  tentatively  adopt  a 
forthcoming  software  coat  estimating  addition  to  the  RCA  PRICE  model  aa 
the  standard  ESD  software  cost  estimating  technique. 

» 

Objective  Of  The  Recommendation,  There  is  nothing  that  can  be  done  to 
reduce  the  estimating  error  due  to  the  random  variable  nature  of  cost. 
Likewise,  until  further  cost  research  is  performed,  little  can  be  done  to 
reduce  the  error  inherent  in  any  of  the  current  estimating  techniques. 
However,  much  of  the  error  in  program  office  estimates  may  be  caused  by 
the  problems  identified  in  the  research  fl  dings.  The  objective  of  recom- 
mending adoption  of  the  RCA  PRICE  model  as  a standard  technique  is  to 
address  this  type  of  error. 

Advantages  Of  A Common  Technique.  Independent  of  what  technique  is 
adopted,  tho  use  of  a common  technique  promises  many  improvements  to  the 
current  softwaro  coat  estimating  process: 

1,  If  estimates  were  produced  from  a common  technique,  managers 
could  then  reasonable  compare  two  estimates  in  making  a doclsion. 

2,  Selecting  a well-founded  technique  would  eliminate  the  use  of 
many  other  techniques  which  aro  suspect  in  their  prodictive  ability.  The 
wd.de  range  of  unsupported  cost  parameters  in  use  would  bo  narrowed. 

3.  A common  technique  could  insure  that  all  estimators  addressed 
the  same  software  coot  dements  and  the  same  software  development  phases  in 
their  estimates.  Each  phase  could  be  explicitly  identified,  and  estimated. 

4.  To  lessen  tho  problems  caused  by  the  disparate  backgrounds  of 
the  estimators,  a minimum  amount  of  training  in  tho  use  of  a common  tech- 
nique might  be  roqulrod.  In  lieu  of  ouch  training,  a common  technique  could 
be  well-defined  and  well-documented  to  minimize  misinterpretations  and  errors 
in  its  application. 
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5.  With  all  ESD  program*  using  the  same  technique,  a wealth  of 
information  would  soon  be  available  to  determine  whether  the  technique  was 
successful  and  to  determine  he*  the  technique  might  be  improved.  The 
current  use  of  many  different  techniques  inhibits  this  type  of  feedback. 

6.  The  common  technique  could  specify  which  point  in  the  range 
of  possible  costs  is  to  be  selected.  No  longer  would  estimates  range  from 
the  lowest  possible  to  the  highest  possible  cost. 

7.  Finally,  adopting  a single  technique  would  aid  the  independent 
estimators  at  ESD.  Instead  f having  to  verify  the  reasonableness  of  many 
different  and  ill-defined  techniques,  the  independent  estimator  could 
simply  have  to  Insure  that  the  common  method  was  properly  applied.  He 
might  then  spend  more  time  and  effort  in  insuring  that  the  parameters 
input  to  the  model  were  reasonable.  Outside  engineering  assistance  to 
verify  the  estimated  number  of  instructions  might  be  obtained. 

The  RCA  PRICE  Model.  The  specific  technique  recommended  for  tentative 
adoption  is  a forthcoming  addition  to  the  existing  RCA  PRICE  model  (RCA, 
1975).  The  PRICE  nodol  was  developed  by  RCA  and  has  been  marketed  to  a 
wide  variety  of  industrial  and  DoD  uuers.  To  date,  the  model  has  primarily 
been  limited  to  the  estimation  of  hardware  cost.  However,  in  the  Fall  of 
1976 » RCA  plans  to  expand  the  model  to  include  the  capability  to  estimate 
the  cost  of  software  development. 

The  software  coat  stimating  addition  to  the  PRICE  model  is,  to  some 
extent,  an  adaptation  of  the  Wolverton  technique  discussed  in  Chapter  III. 
The  technique  first  requires  the  estimator  to  divide  tne  software  into 
modules  and  to  estimate  the  size  of  the  individual  modules.  The  design  is 
required  to  produce  modules  no  larger  than  1000  instructions  in  size.  The 
estimator  is  then  asked  to  determine  the  type,  complexity,-,  and  data  storage 
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requirements  of  each  module  as  well  as  any  similarity  between  the  new  module 
and  past  efforts. 

After  the  modules  have  been  classified,  the  model  provides  "cost  per 
category  of  instruction"  factors  to  estimate  the  cost  of  each  module. 

These  factors  are  based  upon  experiences  at  RCA  for  a particular  category 
of  instruction. 

For  each  software  development  phase  included  in  a particular  program, 
tho  model  insures  that  each  phase  is  explicitly  addressed.  It  also  insures 
that  the  various  elements  of  software  coot  such  as  direct  labor  and  computer 
time  are  also  explicitly  addressed. 

While  the  model  is  quite  detailed,  the  information  and  judgments 
roquired  to  use  the  modol  are  fairly  simple  once  a detailed  software  design 
is  available.  Tho  model  is  an  on-line  computer  model  which  makes  it  simple 
for  the  estimator  to  input  his  information  and  judgments.  It  also  makes  it 
easy  for  an  estimator  to  quickly  determine  the  cost  Impact  of  proposed 
changos  to  tho  softwaro  parameters. 

Despite  tho  promising  aspects  of  the  RCA  PRICE  model,  it  haB  not  yet 
demonstrated  that  it  is  any  better  than  other  techniquos  in  consistently 
producing  accurate  and  reliable  cost  estimates.  However,  the  reason  tor' 
recommending  tho  PRICE  modol  is  not  that  it  has  proven  to  produce  better 
estimates.  Instead,  the  recommendation  is  made  due  to  tho  workable  nature 
of  tho  modol  as  well  as  some  unique  advantages  it  offers, 

RCA  PRICE  Model  - Advantages.  The  RCA  PRICE  model  is  the  result  of  a 
woll  supported  rooearch  effort  by  a major  hardware  and  software  contractor, 
Tt  has  boon  dovoloped  with  tho  best  of  motives  - profit.  However,  there 
are  other  reasons  for  recommending  its  use: 
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1.  The  RCA  PRICE  model  will  be  easily  available  to  all  ESD  pro- 
gram offices.  The  ESD  Cost  Analysis  Division  contracts  with  RCA  for  the 
right  to  use  the  model.  Consultation  services  as  well  as  a complete 
training  program  are  also  made  available  from  RCA. 

2.  The  PRICE  model  is  now  used  to  estimate  hardware  cost  for 

all  ESD  programs.  In  a briefing  by  the  chief  of  the  Cost  Analysis  Division, 
the  PRICE  model  was  said  to  consistantly  produce  estimates  within  10%  of 
the  actual  cost  (Grimm,  1976).  While  this  same  success  might  not  be 
shared  by  the  software  addition  to  the  model,  the  current  success  of  the 
model  as  well  as  its  reputation  merit  its  tentative  adoption. 

3.  While  the  model  requires  a detailed  software  design,  the 
program  offices  at  ESD  have  access  to  adequately  qualified  software  per- 
sonnel who  can  properly  utilize  the  model.  No  sophisticated  knowledge  of 
cost  estimating  or  of  modeling  is  required. 

4.  One  added  advantage  of  the  model  is  that  it  requires  the 
software  to  be  divided  into  1000  Instruction  modules.  By  requiring  thiB 
level  of  detail  in  the  design,  the  model  insures  that  the  operational  user’s 
requirements  have  been  sufficiently  defined  to  support  a detailed  design. 

If  such  a design  cannot  be  made,  a program  manager  might  infer  that 
additional  effort  is  required  to  more  precisely  and  firmly  define  the 
sometimes  elusive  user’s  requirements. 

5.  The  model  will  probably  be  adopted  by  other  DoD  agencies  and 
defense  contractors.  Such  widespread  use  should  result  in  large  amounts  of 
feedback  to  RCA  concerning  the  success  of  the  model.  Improvements  in  pro- 
dictive  accuracy  should  bo  quickly  forthcoming  from  this  feedback. 

6.  Initially,  the  model  will  be  based  on  the  actual  cost  experi- 
ences of  RCA,  a major  and  experienced  developer  of  both  hardware  and  software 
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systems,  The  cost  per  category  of  instruction  factors  will  represent 
values  measured  by  an  intensive  RCA  internal  research  program*  This 

V!*  *■ 

contrasts  to  other  techniques  where  the  sources  of  cost  factors  are  fre- 
quently not  identified. 

?.  A major  advantage  of  the  RCA  PRICE  model  la  that  it  defines 
in  detail  the  various  software  cost  olomonts  and  development  phases.  With 
widespread  use  of  the  model,  those  definitions  may  servo  to  enhance  the 
flow  of  software  cost  information  on  a common  basis. 

The  Recommendation.  There  are  many  problems  inhibiting  the  develop- 
ment of  accurate  and  reliable  estimates  at  ESD.  By  adopting  the  RCA  PRICE 
model  as  a common  technique  many  of  these  problems  can  bo  minimized. 

Therefore,  the  early  adoption  of  the  RCA  PRICE  model  as  the  standard  ESD 
software  cost  estimating  methodology  is  rocommoaded. 

Contractor  Furnished  Cost  Information 

A second  part  of  the  software  cost  estimation  process  which  can  bo 
Improved  is  the  management  of  contractor  furnished  software  cost  information. 

The  contractor  furnishes  ouch  information  in  hlB  initial  cost  proposal  and 
in  his  periodic  Coat  Performance  Reports  (CPR).  A more  uniform  policy 
towards  the  use  of  this  cost  Information  can  significantly  Improve  the 
software  cost  estimating  environment  at  ESD. 

Cost  Proposals.  Finding  //9  indicated  that  not  all  program  offices 
receive  data  which  supports  the  contractor's  initial  estimate  of  software 
cost.  Even  where  this  information  is  furnished,  it  sometimes  is  no'  suffi- 
cient to  enable  the  program  office  to  judge  the  reasonableness  of  the 
estimate.  For  example,  the  software  cost  might  be  based  on  an  average 
productivity  of  eight  source  instructions  per  day.  However,  unless  this 

■f*  v 

factor  is  supported  by  hiatoi'ical  evidence,  the  program  office  cannot  ; [ 
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determine  whethor  tho  clalmod  productivity  and  the  resulting  cost  are 
reasonable. 

Without  the  ability  to  completely  understand  how  tho  contractor 
arrived  at  his  coot  eQtlmato,  two  problemn  con  occur.  The  first  problem  is 
one  of  miounderatanding,  Difficultion  in  accurately  doaoribing  the  work  to 
bo  porformod  aa  vroll  on  tho  abort  time  available  for  tho  preparation  of 
technical  proposals  can  cauao  nlgnlficant  technical  mloundorutandingo  to 
arise  betwoon  tho  Air  Force  and  tho  contractor,  Theae  nlaunderatandinga 
can  have  a negative  impact  upon  both  partloe  and  must  be  avoided. 

Finding  // 10  indicated  that  those  minunderatandingo  do  occur.  In  one 
case,  contractoro*  oatimato  of  tho  olzo  of  a software  effort  ranged  from 
153|000  instructions  to  766,000  instructions.  Contractor  bids  for  the  sane 
software  efforts  ranged  from  a minimum  of  two-to-one  to  a maximum  of 
ssventeon-ty-ono  for  the  three  procurements  atudiod. 

One  approach  to  minimizing  thoae  minundorstandinga  in  to  have  the 
contractor  fully  explain  the  aaaumptiono  ho  made  in  estimating  the  coat  of 
software,  significant  difforoncoa  in  tho  oizo  of  tho  effort  or  in  the 
level  of  difficulty  oxpoctod  would  indlcato  the  nood  for  additional  technical 
discussion  betwoon  tho  technical  roprooontutives  of  tho  Air  Force  and  tho 
contractor,  Tho  cost  oatimato,  and  the  methodology  ueod  in  deriving  it, 
offer  the  boat  vohic'lo  for  determining  :Lf  any  grooo  miaunderotandingB 
exist  betwoon  tho  Air  Force  and  tho  contractor.  However,  to  analyzo  tho 
contractor' a estimato  of  cost,  FS1)  rauot  first  roquiro  that  all  cost  propooals 
for  softwaro  apociflcally  explain  the  method  and  aonumptiohs  used  to  make 
tho  oatimato, 

A second  potential  problem  can  occur  if  the  coot  proposal  does  not 
sufficiently  detail  tho  aoftwaro  coat  oatimato.  The  uocond  problom  is  the 
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"buy-in"  whoro  a contractor  knowingly  oubmite  an  unroallntlcally  low  bid 
i'or  noftwaro,  An  dinouBood  in  Chapter  11,  a buy-in  can  load  not  only  to 
inequitable  contract  awardu,  but  eventually  to  additional  coot  to  tho  Air 
Force  whon  tho  true  coot  o.f  tho  noftwaro  bocomeo  apparent.  Tho  beut  method 
to  avoid  this  problem  la  to  bo  able  to  fully  anulysto  tho  contractor* b 
initJal  ootimato  of  uol'twaro  ooutu,  '.i  f a contractor  claims  that  ho  will 
exporionco  a productivity  of  SO  uouroo  inntructiunu  por  man-day,  thou  tho 
program  office  can  take  action  to  verify  tho  ronnonablonona  of  thio  factor. 
Audita  by  tho  Dol’onno  Contract  Audit  Agonoy  (DOAA)  or  ntudion  by  tho 
bofonao  Contract  Adminlotration  iio).*vicoa  ( )XlA/> ) own  bo  unod  to  verify 
whothor  tho  factor  iu  romiouuhlo  haood  upon  tho  nature  of  tho  now  work 
and  upon  tho  hlntorlual  performance  of  tho  contractor. 

To  avoid  buy-inn  or  mioundorntandingn,  it  in  rooommondod  that  all 
contractor  oont  propouaiH  for  noftwaro  bo  roqutrod  to  fully  support  tho 
noftwaro  coat  ootimato,  Aooumptiono  an  to  tho  ni'/.o  of  tho  offort  or  n«  to 
tho  oxpootod  productivity  nhould  bo  clearly  idontlfiod  and  aupportod  by 
onglnoorlng  and  hlntorlual  data.  With  a complete  and  mutual  undo rn landing 
of  tho  contractor* n initial  on lima to  of  noftwaro  coot,  a program  manager 
can  bo  bottor  uimurod  that  hio  program  I. a off  to  a good  and  nolid  atari. 

Cent  Performance  Sopot'tii,  Finding  H\-'j  explored  tho  oont  information 
oubmittod  by  oontructorn  In  tho. I r periodic  Gout  Performance  Hoportu,  Grono 
Undoroatimatou  of  noftwaro  cent  wore  common,  Thono  groim  orro.ru  were  alwayo 
Blow  to  bo  dlocovorod,  Cclvodulou  nlippod  and  indirect  coat  built  up.  To 
avoid  thono  problomn,  a contractor  not  only  noodo  tho  ability  to  produce 
bottor  initial  outimatoo.  He  alrto  noodn  tho  ability  to  quickly  update  thio 
ootimato  an  actual  coot  and  performance  data  become  available  to  him.  In 
OBBonce  tho  contractor  roqulrorj  a dynamic  estimating  ability  to  quickly 
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aoooaa  tho  not  impact  of  actual  data,  ThiG  dynamic  ability  la  roquirod  to 
permit  oarly  identification  of  software  problomo*  Tho  current  proceoc  ia 
not  adequato, 

Currontly,  tho  contractor  aimply  Booms  to  proparo  hio  initial  oatimato 
and  continuo  to  work  until  tho  actual  oont  makeo  it  blatantly  apparent  that 
tho  initial  oatimato  v/ao  wrong,  Majox.'  changoa  in  tho  initial  ootimato  do 
not  occur  until  it  ia  too  Into  to  prevent  major  uchodulo  ulipo,  Inotoad 
of  tlite,  a dynamic  ability  ia  roquirod  no  that  orrora  in  the  oatimato  aro 
idontifled  quickly. 

To  obtain  tbie  capability,  ooftwax'o  managero  muat  firat  xmderotand  all 
of  tho  unBumptlono  wh.lch  aro  mado  in  b ho  initial  uoftwuro  coat  estimate. 

Por  examplo,  certain  modulon  aro  oxpoctod  to  bo  of  u certain  length  and 
programmer  productivity  in  oxpoctod  to  bo  at  a cortuin  level,  A»  actual 
Hi  so  and  productivity  data  bocomon  availablo,  tho  contx'actor  nhould  bo 
roquirod  to  retux'n  to  kLo  initial  oatimato  to  dotorralno  if  there  worn  any 
major  error u In  hla  aaoumptiona.  If  thoro  wore  major  orrora,  then  tho  ontlro 
aoftwax'o  oatimato  may  need  to  bo  roviood,  A contractor  cannot  oimply  add 
tho  additional  coot  of  one  modulo  which  doublod  in  also.  Ho  muat  detormlno 
if  tho  oarno  doubling  in  a iso  ia  pouelblo  or  probable  for  tho  remaining 
effort. 

To  aid  tho  contx'actor  in  developing  a dynamic  oatimato,  tho  program 
office  ohould  bo  able  to  dovolop  thoir  own  dynamic  oatimato.  Using  tho  RCA 
PRICK  modol,  tho  pi'ogram  office  can  dovolop  an  initial  oatimato  baaod  on 
tho  clearly  idontifiod  aouuraptiono  mado  in  the  contractor* a propoaal,  At» 
actual  data  la  colloctod,  tho  program  office  can  re-run  the  PRICE  model  to 
goo  if  tho  actual  data  cauaea  any  significant  changes  in  tho  total  coat. 

If  significant  differences  aro  identifiod  botwoon  tho  program  office  oatimato 
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and  the  ostimato  in  tho  Coot  Performance  Poport,  quick  management  action 
is  indicated.  Tho  "good  times  are  just  around  tho  cornor"  attitude  must 
be  avoided.  Ae  pointed  out  by  Finding  #13,  things  nevor  got  better  in  a 
■oftware  development.  They  alwaye  got  worse.  With  this  in  mind,  it  is 
essential  that  tho  contractor  develop  a dynamic  ooti  mating  capability  to 
quickly  utilise  the  actual  cost  and  pox'fGxwunce  fuouback  which  la  available. 

The  Pocommondatlon.  Contractor  furnished  ooltware  cost  information 
oan  offer  cignificant  improvements  to  software  management  If  a more  uniform 
and  intensive  policy  in  adopted  towardo  tho  management  and  utilization  of 
thle  information.  It  ia  therefore  recommended  that  all  coet  proposals  for 
eoftwaro  bo  required  to  fully  oxploin  and  support  the  method  and  assumptions 
ueed  in  oetimating  eoftwaro  coot.  In  addition,  tho  eetimatoe  provided  in 
the  Coat  Performance  Hoporte  ehould  bo  dynamic  estlmateu  baoed  on  the  feed- 
back providod  fx'om  actual  coat  and  performance  data. 

Future  Poooarch  Into  Software  Coot  Estimation 

Aa  previously  stated,  tho  1JCA  PRICE  model  may  not  reault  in  reliable 
and  accurate  eatimatee,  While  the  model  doen  offer  promise,  it  offex*s  no 
guaranteeo.  Managers  at  ESD  ehould  not  bo  content  with  oimply  adopting 
the  RCA  PRICE  model.  Further  roeearch  into  this  area  in  esaontial  and 
managers  ehould  insist  that  research  ie  continued  to  hotter  deocrlbe  the 
relatlonohips  that  oxlat  between  software  coot,  software  parameters,  and 
weapon  ayntem  parameters. 

However,  any  further  research  into  this  area  needs  to  be  conducted 
with  strong  direction  and  according  to  a definite  plan.  One  only  has  to 
look  at  tho  many  references  in  thio  roport’o  bibliography  to  appreciate 
that  millions  of  DoD  dollars  must  have  been  spent  in  the  search  for  an 
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accurate  software  cost  estimating  teehniquo » Yot  many  of  the  efforts  aro 
repetitive.  Most  aro  email  efforts  with  large  objectives.  Few,  if  any, 
build  upon  the  efforts  of  other  researchora,  Tho  not  roault  ia  that 
doapite  tho  expenditure  of  mllllona  of  dollars,  tho  past  raaearch  efforta 
have  not  resulted  In  the  development  of  a technique  which  is  reliable  or 
accurate,  Littlo  If  any  uoo  is  mado  by  today’s  program  managers  of 
yeotorday’o  reaoarch  into  software  cost  estimating.  While  the  failure 


of  past  efforta  may  havo  many  cauaoa,  some  seem  apparent.  Moot  of  the 
efforts  woro  conducted  on  an,  individual  basis,  sponsored  by  different 
agenclea,  with  little  correlation  or  cooperation  betweon  the  efforta. 


To  insure  that  futuro  research  efforts  do  not  simply  result  in  more 
reports  and  longer  bibliographies,  research  in  the  area  must  be  strongly 
guidod  by  a well-foundod  plan.  While  this  type  of  reamarch  might  best  bo 
accomplished  by  a ronoarch  labox'atory  such  as  RADII,  tho  direction  of  the 
effort,  should  bo  contorod  around  tho  needs  of  tho  product  divisions  suoh 
as  BSD , Most  of  the  data  required  for  software  coat  research  in  currently 
being  collected  by  those  weapon  system  programs  having  major  noftware 
acquisitions.  Any  research  laboratory  must  rely  strongly  on  this  typrt  of 
data.  While  tho  orront  data  might  not  bo  uniformly  collected,  guidance 


I 


from  a laboratory  could  bo  uaod  to  standardize  tho  typo  and  levol  of  soft- 
ware coat  infox'mation  collected  from  contractors.  With  commc  n coot  olemonte 
and  common  software  development  phasoa,  reaearchore  would  finally  bo  able 
to  amass  enough  reliable  data  to  reasonably  analyze  the  relationship 
between  cost  and  other  parameters. 


m . 


In  summary,  futuro  research  efforts  are  vitally  necessary,  Howevor, 
any  further  research  efforts  should  be  strongly  guidod  by  a research  plan. 
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The  .roawarch  laboratory  and  the  product  division  nhould  cooperate  to  tho 
fulleat  extent,  The  laboratory  is  dependent  upon  tho  product  dlvloion  to 
provido  well-def Anod  software  coot  data  on  actual  aoftwaro  ttequAnltlows. 

Tho  product  divio.I.on  1b  likewise  dopondont  upon  tho  laboratory  to  eventually 
provido  an  Improved  mothod  for  ootlmatlng  tho  cont  of  aoftwaro,  Wox’ld.n g 
togother  Jointly,  future  offortu  can  provido  tho  information  nuoounary  to 
devolop  tho  accurate  and  roliablo  ooftwaro  oout  outimatou  roqulx'od  fox'  good 
software  management. 


Conclusion 

In  roue«rahing  tho  aroa  of  « oftwaro  coot  ontiraution,  It  wa«  admittedly 
easier  to  rxoroly  oxploro  how  ontimaton  ware  currently  being  prepared,  Tho 
more  difficult  and  more  fruitful,  x'oaoaroh  X'omainn  - to  develop  a butter 
software  coot  oMtimatAng  toolmique,  However,  thin  ronoaroh  han  hope  fully 
accompli*;  hod  it«  objoctivo  of  providing  managont,  ronoaroharn,  and  coot 
oatlmatorn  with  an  Ancroanod  innight  into  tho  problomo  of  the  uoftware  uoot 
estimating  proconu.  This  Insight  may  npeed  tho  development  of  hotter 
estimation  techniquou. 

However,  thox-e  in  ono  action  which  can  hotter  improve  the  eutimation 
of  eoftware  than  tho  rocommondutionu  mado  by  thin  effort  or  by  any  further 
renoarch  effort.  To  improve  /joftwaro  coot  ootlmation,  noftwaro  manugorn 
muot  rocogtiizo  tho  vital,  importance  that  noftwaro  coot  outimatou  havo  in 
many  eoftwax'e  managomont  docifliona,  Thooo  ontimutou  aro  not  nimply  put 
together  to  Batin fy  tho  x'oquiromontu  of  a budgot,  I notoad,  tho  outiraato 
becomoa  v o primax’y  docinlon  cx'ltex'ia  in  almout  ovory  major  noftwaro  munugo- 
ment  decision,  If  dominion  makeru  want  to  improve  tho  managomont  of  ooft~ 
wuro  acqxiinltion,  thon  thoy  muot  rocognlzo  tho  importance  that  an  accurate 
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and  reliable  estimate  has  to  the  decision  process,  Onco  managers  rocogniae 
thie,  the  increased  effort  and  increased  attantlon  will  probably  result  In 
greater  improvement  to  tha  aoftwara  coat  eetimating  process  than  any  othar 
notion, 

laproving  aoftwara  ooat  estimates  proaiaaa  to  improve  aoftwaro  manage- 
want,  However,  tha  roadar  ahould  remember  that  cost  estimation  ia  only  ono 
aaall  facet  of  tha  aoftwara  aanagaaant  environment,  If  we  are  to  greatly 
Improve  our  ability  to  manage  software,  than  wa  need  to  oontinua  raaaaroh 
in  many  othar  critical  areas  auch  aa  aoftwara  productivity,  aoftwara  con- 
figuration control,  and  aoftwara  milestones.  Coat  aatlmating  ia  only  one 
small  part  of  tha  management  problem. 

In  conclusion,  this  raaaaroh  has  identified  some  problems  which  exist 
at  BSD  and  may  exist  at  othor  Dol)  aoftwaro  acquisition  activities,  Xt  is 
hoped  that  tha  recommendations  resulting  from  this  research  effort  are 
favorably  considered  and  adopted  by  all  such  activities.  Their  adoption, 
coupled  with  a greater  awareness  of  the  problem  area  and  further  research, 
promise  to  improve  tho  management  of  software  acquisition  within  Dol), 
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Appendix  A 


ACCl/Capt  Bourdon/5 2 2 7/MD/2  Apr  76 


ACC  7 April  1976 

Software  Cost  Estimation 


DCB 

DCM 

MCN 

OCN 

WWE 

DCJ 

DCY 

MCV 

ocs 

XRI 

DCK 

FAE 

OCD 

ocu 

YSX 

DCL 

FAM 

OCL 

OCW 

YWX 

1.  Software  cost  estimation  is  a difficult  and  critical 
process.  In  order  to  provide  some  insight  into  the 
problems  of  software  cost  estimation,  Capt.  Tom  Devenny, 
an  AFIT  student,  is  performing  a research  effort  to 
describe  and  study  the  software  cost  estimation  process 
at  ESD.  This  research  effort  should  provide  program 
managers  a better  understanding  of  software  cost  estimat- 
ing and  aid  software  cost  estimators  in  preparing 
estimates. 


2.  Your  support  of  this  research  effort  is  requested. 
Capt  Devenny  will  be  conducting  interviews  at  ESD  from 
12  April  through  23  April.  Request  you  establish  a 
point  of  contact  for  this  interview.  The  point  of 
contact  should  be  the  person  or  persons  whom  you  believe 
to  be  most  knowledgeable  of  the  software  cost  estimate (s) 
made  for  your  program. 


3.  Request  the  name(s)  of  your  point  of  contact  be 
forwarded  to  ESD/ACCI,  Capt  Bourdon,  by  9 April. 


RICHARD  W.  GRIMM,  Major,  USAF 
Chief,  Cost  Analysis  Division 
Comptroller 


ACCI 
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Appendix  D 

SOFTWARE  COST  ESTIMATION  QUESTIONNAIRE 
PRIVACY  STATEMENT 


In  accordance  with  paragraph  30,  AFR  12-35*  the  following  information 
is  provided  as  required  by  the  Privacy  Act  of  1974: 

a.  Authority: 

(1)  10  U.S.C.,  80-12,  Secretary  of  the  Air  Force,  Powers,  Duties. 

Delegation  by  Compensation;  and/or  — - 

(2)  EO  93-97,  22  Nov  43,  Numbering  System  for  Federal  Accounts 
Relating  to  Individual  Persons;  and/or 

(3)  DOD  Instruction  1100.13,  17  Apr  68,  Surveys  of  Department  of 
Defense  Personnel;  and/or 

(4)  AFR  178-9*  9 Oct  73,  Air  Force  Military  Survey  Program. 

b.  Principal  purposes.  This  survey  is  being  conducted  to  collect 
information  to  be  used  in  research  aimed  at  illuminating  and  providing 
inputs  to  the  solution  of  problems  of  interest  to  the  Air  Force  and/or 
DoD.  Specifically,  this  survey  aims  to  provide  managers  and  software 
cost  estimators  greater  insight  into  the  area  of  software  cost  estimation. 

c.  Routine  uses.  The  survey  data  will  be  converted  to  information 
for  use  in  the  research  of  software  cost  estimation  problems.  Results  of 
the  research,  based  on  the  data  provided,  will  be  Included  in  a written 
master's  theBis  and  may  also  be  Included  in  published  articles,  reports, 
or  texts.  Distribution  of  the  results  of  the  research,  based  on  the 
survey  data,  whether  in  written  form  or  presented  orally  will  be  unlimited, 

d.  Participation  in  this  survey  is  entirely  voluntary, 

e.  No  adverse  action  of  any  kind  may  be  taken  against  any  individual 
who  elects  not  to  participate  in  any  or  all  of  this  BUrvey. 


Interviewer:  Capt  Tom  Devenny 


Date: 

Time: 

Place: 
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1.  Briefly  deocribe  the  aoftwuro  acquisition  associated  with  your 
program.  What  is  being  acquired?  When?  How? 


2,  Describe  how  tho  program  offico  estimate  of  software  cost  was 
obtained. 


3,  Briefly  describe  your  educational  and  work  background. 
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4.  How  many  y© ixrn  have  you  boon  involvod  in  aystema 

acquisition?  . f of  yeans 

5.  How  many  of  thoso  years  havo  you  boon  involvod  in 

the  acquisition  of  computer  software?  $ of  years 

fi,  How  many  college  credit  hours  of  software  related 
courses  havo  you  attended? 

7,  Have  you  over  worked  as  a programmer? 

How  long? 

8,  Havo  you  over  directly  supervised  a group  of 
programmers? 

How  long? 

9,  Have  you  ovor  had  any  formal  training  in 
cost  estimation? 

10,  Havo  you  over  estimated  the  software  coot 
of  other  projocto? 

11,  At  the  timo  that  the  progrum  offico  made  its  first  formal  (o.g. 
Program  Managomont  Plan)  estimate  of  software  development  coats, 
how  voll  known  wore  the  following  factors? 

a.  Type  of  computor  (o.g,  UYK-7,  UNIVAC  1108,  otej 

Definitely  known  Gone rally  known  , Slightly  known  Unknown 

b.  Configuration  and  typos  of  poriphoralo 

Definitely  known  Gonorally  known  ^Slightly  known  Unknown 

c.  Memory  and  storage  n\zo 

Definitely  known  Gone rally  known  !> lightly  known  Unknown 

d.  Operating  system 

_ ^Definitely  known  Gonorally  known  Slightly  known  ^Unknown 

o.  Compiler  and/or  assembler 

Dofinitoly  known  ^Gonorally  known  Slightly  known  Unknown 

f,  Numbor  and  typo  of  intorfaooe 

Definitely  known  Gonorally  known  Slightly  known  Unknown 

g,  Numbor  of  input  moasago  typos 

Definitely  known  Gonorally  known  Slightly  known  Unknown 


ft  of  credit  hours 

Yoa mNo 

jt-  of  yoax'o 


Yes  No 

# of  years 


Yes  No 


Yoa  No 


132 


r lili  Ii> 


— i*rwe‘'* 


GJSM/SM/76C-4 


h,  Number  of  output  message  types 

Definitely  known  Generally  known  Slightly  known  Unknown 

1 « Response  time  requirements 

Definitely  known  Generally  known  Slightly  known  Unknown 

j.  Number  of  Computer  Program  Configuration  Itema  (CPCIs) 

Definitely  known  Generally  known  Slightly  known  Unknown 

V2,  At  the  time  the  contract  was  awarded,  how  woll  known  were  tho 
following  factors? 


a.  Type  of  computer  (e,g,  UYX-7,  UNIVAC  1108,  etc.) 

Definitely  known  Generally  known  —jSllghtly  known  Unknown 

b.  Configuration  and  types  of  peripherals 

Definitely  known  Generally  know  Slightly  known  Unknown 


c.  Memory  and  storage)  size 

^Definitely  known  Generally  known  Slightly  known  Unknown 

d.  Operating  system 

Definitely  known  Generally  known  Slightly  known  Unknown 

o.  Compiler  and/or  assembler 

Definitely  known  Generally  known  Slightly  known  Unknown 

f.  Number  and  type  of  interfaces 

Definitely  known  Qenorally  known  Slightly  known  Unknown 


1?. 


g.  Number  of  input  message  types 

^Definitely  known  Generally  known  Slightly  known  Unknown 

h.  Number  of  output  message  typCB 

Definitely  known  Generally  known  Slightly  known  Unknown 

i.  RoBponeo  time  requirements 

Definitely  known  ^Generally  known  Slightly  known  Unknown 

j.  Number  of  Computer  Program  Configuration  Items  (CPCIs) 

Definitely  known  ^Generally  known  Slightly  known  Unknown 

Are  you  familiar  with  the  concept  of  a "management  reserve"?  Yes  No 
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14.  Did  the  program  office  establish  a management  reserve 

for  software?  Tee  Wo 

15.  If  yes,  was  the  management  reserve  left  in  the  program 

office  budget  until  the  contract  was  essentially  complete?  YeB  Wo 

16.  Do  you  feel  that  a software  management  reserve  should  be 
established  for  oach  major  software  acquisition? 

Strongly  agree  Agree  No  opinion  ^Disagree  Strongly  disagree 


17*  Do  you  feel  that  such  a management  reaorvo  would  be 

approved  during  the  budget  process  at  HQ  AFSC  and  Yes  No 

HQ  USAF?  No  opinion 

18.  What  size  of  a management  reserve  (as  a percentage  of 
the  estimated  software  cost)  do  you  feel  a program 
similar  to  yours  should  budget?  % 

19*  Was  an  independent  cost  estimate  prepared  for  software?  Yes  No 

20.  What  was  the  Independent  estimate?  < 

21.  What  was  the  program  office  estimate  at  that  time?  $ 


22.  Which  ostimate  do  you  feel  is  more  accurate?  SPO  Independent 

No  opinion 


2 3.  Did  you  provide  the  independent  estimator  with  your 

estimate  of  the  number  of  computer  instructions?  Yes  No 

24,  Briefly  describe  the  typo  of  information  you  provided 
to  the  independent  estimator. 


25.  Was  the  accuracy  of  the  SP0*e  software  cost  estimate 

challenged  by  anyone?  Yes  No 

26.  If  yes,  by  whom?  (Please  list) 
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27. 

Was  the  accuracy  of  the  contractor' a software  cost 

estimate  challenged? 

Yes No 

28. 

If  yea,  by  whoa?  (Please  list) 

29. 

What  was  the  program  office  estimate  of  software 
cost  prior  to  contract  award? 

& 

30. 

What  was  the  contractor's  estimate  of  software  cost 
in  his  proposal? 

S 

31. 

How  low  do  you  believe  the  actual  software  cost  might 
eventually  be? 

$ 

32. 

How  high  do  you  believe  softrare  cost  might 
eventually  be? 

5 

33. 

What  percentage  of  the  contract  period  is  complete? 

% 

34. 

Which  of  the  following  contractor  tasks  were  included 
in  the  program  office  estimate  of  software  coat? 

a. 

Analyze  user  requirement 

Yes 

No 

b. 

Prepare  system  specification 

Yes 

No 

c. 

Define  system  interfaces 

Yes 

No 

d. 

Design  the  data  base 

_ _JTes 

l.’O 

e. 

Develop  program  test  plans 

Yes 

No 

f. 

Specify  all  input  and  output  message 

formats 

Yes 

No 

K. 

Design  and  flow  chart  each  computer  ; 

program 

component 

Yes 

No 

h. 

Write  coded  program  statements 

Yes 

No 

i. 

Compile  and  check  program  code 

Yes 

No 

j. 

Plan  and  run  functional  test  of  each 

program 

Yes 

No 

k. 

Plan  and  conduct  integration  test 

Yes 

No 

1. 

Train  UBer  personnel 

Yes 

No 

m. 

Conduct  demonstration  test 

Yes 

No 
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n.  Assist  in  operational  shakedown 

o.  Develop  software  maintenance  plan 

p.  Maintain  software  after  delivery 

35.  Did  the  contractor's  cost  proposal  indicate  how  the 
cost  of  software  was  estimated? 

56*  If  yes,  briefly  describe  the  method  the  contractor  uaod 
to  support  his  software  coat  ootimato. 


57,  Was  the  contract  for  software  awarded  competitively? 

38.  What  was  tho  lowest  softwaro  cost,  contained  in  any  of 
the  contractor  proposals  (i.e.  both  successful  and 
unsuccessful  biddsro)? 


39.  What  was  the  highest  software  coot  contained  in  any  of 
the  contractor  proposals  (i.e.  both  successful  and 
unsuccessful  bidders)? 

40,  Does/will  your  contract  call  for  the  reporting  of 
cost  data? 


p 


41,  Doee/will  the  contractor  report  the  cost  of  softwaro 
as  a separately  identified  item? 

42,  Does/will  the  contractor  report  the  total  number  of 
computer  instructions? 

43,  Does/will  tho  contractor  report  the  total  number  of 
computer  instructions  for  each  Computer  Program 
Configuration  Item  (CPCX)? 


44.  Doee/will  the  contractor  report  tho  total  number  of 
direct  man-hours  charged  to  software? 


45.  Does/will  the  contractor  report  tho  total  number  of 
direct  man-hours  for  each  CTCI? 
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46.  Does/will  tho  contractor  roport  tho  number  of  macldno 

hour*  requlrod  for  software  development  and  test?  Yea  Wo 

47.  Doea/will  the  contractor  report  tho  number  of  aaohlne 
hours  required  for  the  development  and  teat  of  each 

CPC1?  Yea  No 

48.  What  was  the  contractor’s  budgeted  coat  for  software 
at  tho  following  points  after  contract  award? 

Six  munthuV  It 

Twelve  months?  it 

Bightoen  months?  8 

Twenty- four  months?  8 

49.  What  was  tho  contractor’ a actual,  coat  for  software 
at  tho  following  points  after  contraot  award? 

Six  months?  8 , 

Twelvo  months?  8 , 

Rightoon  months?  8 

Twenty-four  months?  8 

90.  What  was  tho  percent  oompleto  reported  for  noft- 
ware  at  tho  following  points  after  contract  award? 


Six  months?  8. 
Twelve  months?  8, 
Kighteen  months?  8, 
Twenty- four  months?  8, 


51.  What  was  tho  program  office  estimated  cost  of  software?  ti 

52.  What  was  the  software  cost  estimate  in  the  contractor’s 

proposal?  8 

53.  What  was  cho  software  cost  ootimato  at  tho  Dystora 

Design  Review?  8 

54.  How  many  months  aftor  contract  award  was  tho  Dystom 

Design  Review  held?  M of  months 

55.  What  was  tho  software  cost  estimate  at  the  Preliminary 

Design  Review'/  8 

56.  How  many  months  aftor  contract  award  wae  tho 

Preliminary  Deeign  Review  hold?  ft  of  months 
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Appendix  C 


Ratio  of 
Katiaatid/ 
Actual  Coat 
to  Filial  Coat 


Huabar  of  Hontha  Aftar  Contract  Award 


PROGRAM  )3  - Coat  Ptrforaanot  Data 
Fiaura  7 
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Ratio  of 
Estimated/ 
Actual  Cost 
to  Final  Coat 


Estimated  Coat 
Final  Coat 


Actual  Cost 
Final  Coat 


3 6 9 12  15  18  21  24  2? 


Huaber  of  Months  After  Contract  Award 


PROGRAM  2 - Coat  Performance  Data 
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Thousands 
of  Dollars 
(*  K) 
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Thousands  of 
Dollars 
(*  K) 


Estimated  Cost 


Actual  Cost 


Nuaber  of  Months  After  Contract  Award 


PROGRAM  q - Cost  Performance  Data 
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